Stillkill/sync_replay_project
1
0 Fork 0
forked from Foxyz/sync_replay_project
Code Pull Requests Activity

Protocol 11: split-track relay (YouTube video+audio), chunked upstream reads

Browse Source 
Relayed media can carry 2-4 byte tracks (video, audio): yt-dlp picks a
split pair when no single-file format exists (h264 first), the source
channel serves each track, the server keeps a buffer per track (budget
split by size, 8 MiB floor) and viewers open an mpv edl:// of the track
URLs. Expired URLs re-resolve once for all tracks. Upstream GETs are cut
into <= http_chunk_size (10 MiB) ranges: YouTube throttles larger ones.
Live streams and DRM are refused; HLS/DASH pieces are a follow-up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
foxyzandClaude Opus 5.5 committed 2026-10-09 19:59:41 +03:00
1 parent b2b77a6b3f
commit b038706dff
19 files changed
+945 -286
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/ARCHITECTURE.md
+61 -24
View File
@@ -395,24 +395,49 @@ What gets relayed (`SessionCoordinator.swapPicked`):
| Link, "Through me" | relayed from that member's computer; for links only they can open |
| Jellyfin item (browser pick or pasted link) | own account first, then the server's login (`/j`), then relayed from the member who put it up (with their sign-in, or the pasted link's key, which stays on their machine) |
Relayed links are range-read as one file: a media file as is, a web page
as yt-dlp's single-file format, resolved where the bytes are fetched (see
"yt-dlp"); the swap carries only the page URL and the mode.
Relayed links are range-read: a media file as is, a web page as yt-dlp's
formats, resolved where the bytes are fetched (see "yt-dlp"); the swap
carries only the page URL and the mode. A page's formats are one file, or
a video-only and an audio-only file (YouTube has only those): a media of
two **tracks**, each relayed as its own byte stream with its own buffer
and URL, played together by viewers as an mpv EDL
(`edl://!no_clip;%n%<video>;!new_stream;!no_clip;%n%<audio>`,
`relayViewerUrl` / `mpvEdl`). media_kit opens an `edl://` URL unchanged
(it writes it to a playlist and `loadlist`s it) and mpv keeps both
streams in sync across seeks; one URL keeps the app's "one media URL"
path (sync, swap, reopen) as is. The source member plays the page
itself (yt-dlp hook) like any other link.
How bytes move:
- The source dials `/r/.../source` (outbound, so NAT/CGNAT are fine) and
answers the server's pulls from the file or the URL (`RelaySource`,
`FileRelayBytes`, `HttpRelayBytes`, `LinkRelayBytes`; a Jellyfin source sends its own
`Authorization` header from this machine). Viewers only ever talk HTTP
to the server, so the source never sees mpv's request pattern.
- The server keeps one `RelayBuffer` per relayed media: 1 MiB slices in
answers the server's pulls from the file or the URL (`RelaySource` over
a `RelayMedia`: its `RelayTrack`s, each a `RelayBytes`:
`FileRelayBytes`, `HttpRelayBytes`, `LinkRelayMedia` / `PageRelayMedia`
for links; a Jellyfin source sends its own `Authorization` header from
this machine). Viewers only ever talk HTTP to the server, so the source
never sees mpv's request pattern. The tracks of a media share the
channel and its flow control (they compete for the same uplink).
- Upstream GETs (`HttpRelayBytes.read`) are split into back-to-back
ranges of at most yt-dlp's `http_chunk_size` for the format, else
10 MiB (`relayUpstreamChunkBytes`); a buffer pull stays one read.
YouTube serves a range up to 10 MiB at full speed but throttles a longer
or open-ended one to about twice the bitrate (measured 0.8 MB/s against
8.6 to 12.5 MB/s); chunked, a 1080p relay seeks in under 0.8 s (4 to
6.6 s unchunked) and even beats mpv reading YouTube directly, since the
buffer reads ahead at full speed.
- The server keeps one `RelayBuffer` per track of a relayed media (one
for a file): 1 MiB slices in
memory, filled by one sequential pull at a time. Kept: the first and
last 2 MiB (container indexes; mpv reads both on open and on seeks),
64 MiB behind the slowest viewer and 256 MiB past the furthest, the
same as each viewer's mpv buffer (`PlaybackController`), so a skip
inside the viewers' buffer window is served from memory. Evicted by
distance from the viewers, never LRU. With 2+ viewers the source sends
each byte about once.
each byte about once. Several tracks split the ahead and behind budgets
by size (one duration, so by bitrate; `relayBudgets`), at least 8 MiB
each, which keeps both windows at the same playback distance (1080p
video + aac: about 246 + 10 MiB ahead); each pins its own head and
tail. A viewer is its token in every track's buffer.
- Filling: a viewer blocked on a slice first (oldest wait first, handing
the source over at slice boundaries), then the pinned slices, then
read-ahead from the slowest viewer. Nothing is fetched while nobody
@@ -431,7 +456,7 @@ How bytes move:
stop. A crash frees it with the process.
- Source away: viewers keep playing from the buffer; a request waits up
to 60 s for missing bytes. The source reconnects with backoff (same
token after a resume); a different file size drops the buffer.
token after a resume); a different size drops that track's buffer.
`state.media.relay.live` follows the channel, and viewers show "waiting
for <name> to pass the video on" while it is down.
@@ -481,10 +506,17 @@ all three link modes. Nothing is bundled or installed by hand.
`~/.cache/watch-together/yt-dlp`.
- **Resolve** (`YtDlp.resolve`): `yt-dlp -J --no-playlist --no-warnings
--ignore-config`, 25 s timeout: title, uploader, duration, thumbnail,
and the best single-file format (`pickSingleFile`: video and audio not
`none`, plain http(s), a video container, tallest up to 1080p) with its
URL and `http_headers`; none when the page has only split or HLS
streams (YouTube today).
live or not, and what a relay reads (`pickTracks`), each format with
its URL, `http_headers` and `http_chunk_size`. Only plain http(s)
formats without DRM count:
- the best single file (`pickSingleFile`: video and audio not `none`, a
video container, tallest up to 1080p, else the shortest above), unless
- a video-only + audio-only pair is taller (`pickSplit`): video tallest
up to 1080p (else the shortest above), avc1 before vp9 before av01
(hardware decoding), then bitrate; audio the original language, not a
`-drc` variant unless alone, m4a/aac before opus, then bitrate.
YouTube has only these (299 + 140 for a 1080p60 video).
- Neither (only HLS/DASH playlists) or a live stream: nothing to relay.
- **Probe** (`probeLink`, `link_probe.dart`): a `Range: bytes=0-0` GET; a
video/audio type, or a generic one on a media file name, is a file
(`looksLikeMediaFile`); anything else goes to yt-dlp, and when yt-dlp
@@ -498,17 +530,22 @@ all three link modes. Nothing is bundled or installed by hand.
(`script-opts=ytdl_hook-ytdl_path=...`, `ytdl-format=bv*[height<=1080]+ba/b`,
once per player), waiting for the download unless the URL names a media
file. A hook error shows as "can't open this link: ...".
- **Through me**: the putter's probe must give a single-file format, else
the swap is refused with a hint to use "Everyone fetches it". Its relay
source reads that format with yt-dlp's headers (`PageRelayBytes`); on
403/410 (expired format URL) it resolves the page again once and goes
on from where it stopped.
- **Through me**: the putter's probe must give tracks, else the swap is
refused ("Live streams can't be passed on", or no video file: only
streaming playlists) with a hint to use "Everyone fetches it". The swap
names the tracks; its relay source reads them with yt-dlp's headers
(`PageRelayMedia`). On 403/410 (expired format URLs, 6 h on YouTube) from
any track it resolves the page again once for all tracks, finds the
same format ids with the same sizes (else "the video changed on the
page") and goes on from where it stopped. Format URLs are bound to the
resolving machine's address, so they never leave it.
- **Server fetches it**: the server probes the link itself
(`LinkRelayBytes`) with its own yt-dlp (off with `--no-yt-dlp` /
`ytDlp: false`) and relays the single-file format the same way. The
putter only refuses a page its own yt-dlp says has no single file (the
server may reach what the putter cannot); when the server cannot read
it, the putter gets an `err` with the reason. Viewers open a fetched
(`LinkRelayMedia`) with its own yt-dlp (off with `--no-yt-dlp` /
`ytDlp: false`) and relays its tracks the same way, setting
`relay.tracks` with `live`. The putter only refuses a page its own
yt-dlp says cannot be relayed (the server may reach what the putter
cannot); when the server cannot read it, the putter gets an `err` with
the reason. Viewers open a fetched
link's `/r/` URL only once `relay.live` is set ("the server is getting
the video…" until then): resolving takes seconds and media_kit's mpv
gives up on a silent request after 5 s (`network-timeout`).
docs/PROTOCOL.md
+43 -13
View File
@@ -1,4 +1,4 @@
# Protocol (proto = 10)
# Protocol (proto = 11)
Wire format only. Behavior (gate, clock, timeouts) is in
[ARCHITECTURE.md](ARCHITECTURE.md). Source of truth:
@@ -30,7 +30,7 @@ with different `proto` refuse each other at `hello` (`proto mismatch`).
## Client to server
```
{"t":"hello","proto":10,"secret":"","user":{"name":"","avatar":""},
{"t":"hello","proto":11,"secret":"","user":{"name":"","avatar":""},
"media":{"name":"","size":0,"duration":0,"url":null},"hostKey":"...",
"resume":"<token>"}
{"t":"ready","ready":true,"source":"manual"} source: manual | auto
@@ -218,10 +218,20 @@ flow control, cases) is in [ARCHITECTURE.md](ARCHITECTURE.md), "Relay".
```
swap media.relay = {} the swapper's machine serves the bytes
media.relay = {"tracks":[{"kind":"video"},{"kind":"audio"}]}
the same, as two tracks played together
media.relay = {"fetch":true} the server reads media.url itself
state media.relay = {"source":"m2","live":true} or {"fetch":true,"live":true}
media.relay = {"fetch":true,"live":true,"tracks":[{"kind":"video"},{"kind":"audio"}]}
```
Tracks: a relayed media is one file, or several byte tracks (a page's
separate video and audio files) that viewers play as one. `tracks` lists
them in order, 2 to 4 entries of `kind` `video` or `audio`; absent (or
anything else) means one file. Every track is a Range-readable byte
stream; other kinds of track (e.g. segmented playlists) would add a field
to the entry.
- `swap`: `relay` with `jellyfin` keeps the ref (the swapper serves it to
viewers without an account; `fetch` is ignored there). `relay` without
`jellyfin`: no URL is stored or broadcast; `fetch` needs an http(s)
@@ -231,19 +241,32 @@ state media.relay = {"source":"m2","live":true} or {"fetch":true,"live":true}
for anyone). A web page `url` is read through the server's yt-dlp; when
the server cannot read the link, the swapper later gets `The server
can't fetch this link: <reason>`. Any later swap ends the relay.
`tracks` on a member's swap names the tracks its source will serve; the
server ignores it with `fetch` and with `jellyfin`.
- `state.media.relay.source`: member id of the source (absent with
`fetch`). `live` (omitted when false): bytes can flow (the source's
channel is connected, or the server could read the link).
channel is connected, or the server could read the link). `tracks`: as
the swap named them, or, with `fetch`, set once the server has read the
link (together with `live`).
- `welcome.mediaUrl` for relayed media without `jellyfin`: this member's
`/r/` URL; absent for the source itself (it plays its own copy). Clients build it after a swap like the `/m` one:
`/r/` URL; absent for the source itself (it plays its own copy) and for
a fetched link the server has not read yet (clients open it on `live`).
Clients build it after a swap like the `/m` one:
`http://<host>/r/default/<token>/<mediaId>/<name>` (name with `/` and
`\` made `_`, then URL-encoded; the server ignores it).
- `GET /r/<sid>/<token>/<mediaId>/<name>`: the media, Range table as for
`/m`. Errors: 401 bad token, 403 kicked, 404 not the current relayed
`\` made `_`, then URL-encoded; the server ignores it). With `tracks`,
it is an mpv EDL of every track's URL
`http://<host>/r/default/<token>/<mediaId>/<i>/<name>`, each framed by
its byte length:
`edl://!no_clip;%<len>%<track 0 url>;!new_stream;!no_clip;%<len>%<track 1 url>`.
- `GET /r/<sid>/<token>/<mediaId>/<name>`: the media (one file), Range
table as for `/m`. `GET /r/<sid>/<token>/<mediaId>/<i>/<name>`: track
`i` (from 0) of a media with `tracks` (track 0 of a one-file media
too). Errors: 401 bad token, 403 kicked, 404 not the current relayed
media (a player still on an older `mediaId` gets 404, never the new
bytes), 410 unknown session, 503 + `retry-after` the source never
connected (60 s). A request whose bytes do not arrive within 60 s ends
early. Bytes count toward `bytesServed`.
bytes) or no such track (the one-file URL of a media with `tracks`), 410
unknown session, 503 + `retry-after` the source never connected or the
server could not read the link (60 s). A request whose bytes do not
arrive within 60 s ends early. Bytes count toward `bytesServed`.
- `GET /r/<sid>/<token>/source` (WebSocket): the source's relay channel,
dialled by the source (works behind NAT). Only the current source's
token is accepted; anyone else, or the source after the next swap, is
@@ -251,15 +274,22 @@ state media.relay = {"source":"m2","live":true} or {"fetch":true,"live":true}
Frames:
```
source -> server {"t":"info","size":<bytes>} first frame
source -> server {"t":"info","size":<bytes>} first frame, one file
{"t":"info","tracks":[{"kind":"video","size":<bytes>},
{"kind":"audio","size":<bytes>}]}
first frame, tracks
binary: [u32 big-endian pull id][bytes] in order per pull
{"t":"end","id":1} {"t":"fail","id":1,"reason":"..."}
server -> source {"t":"pull","id":1,"start":0,"end":1048575} inclusive
server -> source {"t":"pull","id":1,"track":0,"start":0,"end":1048575}
inclusive; no track: 0
{"t":"ack","id":1,"n":65536} bytes arrived
{"t":"cancel","id":1} stop that pull
```
A `size` different from the one before (a reconnect) drops the buffer.
Pull ids are unique across the tracks of a connection; `track` is sent
only for a media with `tracks`. An `info` whose track count differs
from the swap's `tracks` closes the channel with 4001. A track `size`
different from the one before (a reconnect) drops that track's buffer.
## Join and reconnect
docs/ROADMAP.md
+20 -4
View File
@@ -15,10 +15,26 @@ off by deleting it in the same commit as the code. Wire changes also edit
countdown). Later: tracking for file series.
3. **yt-dlp follow-ups** (see ARCHITECTURE "yt-dlp"): check that the
bundled Windows libmpv has the ytdl hook (Lua); if not, resolve in the
app and open the single-file format. YouTube: yt-dlp wants a JS runtime
(deno) for full extraction, and it only has split streams, so relaying
it would need muxing. Formats that need yt-dlp's cookies.
bundled Windows libmpv has the ytdl hook (Lua) and plays `edl://` URLs
through media_kit; if not, resolve in the app and open the relay
tracks' formats. YouTube: yt-dlp wants a JS runtime (deno) for full
extraction. Formats that need yt-dlp's cookies.
- Relay "pieces" tracks for pages with only HLS/DASH (`m3u8_native`,
`http_dash_segments`): a track served as a synthesized VOD media
playlist whose entries are pieces (one segment, init `EXT-X-MAP` or
AES-128 key = one upstream GET, served whole, Range inside a piece
honoured), with a piece buffer per track (pieces by index, filled
like `RelayBuffer`: a blocked viewer, the init piece, then ahead of
the slowest viewer; evicted by distance); `EXT-X-BYTERANGE` folded
into pieces (it also avoids mpv's seek hang on byte-range fMP4
playlists); yt-dlp's DASH `fragments` turned into the same playlist,
so no MPD parser. Wire: a track entry gains `"as":"pieces"`, `info`
lists piece durations, a pull names a piece. Refuse DRM
(`skd:`/SAMPLE-AES/Widevine). Measured in a spike: piece relay of
Mux/Apple/Akamai HLS/DASH opened in 1.5 to 2.2 s, seeks 0.02 to 2.4 s.
- Live streams through the relay (today refused: the sync model needs a
fixed duration and seeking).
- `tool/relay_e2e.dart`: a two-track case over a shaped source link.
4. **AniList account**: sign in (OAuth via a localhost callback), "My list"
filter on the Schedule; today's "Following" (AnimeFlow's monitored