Read once: a link passed on is fetched from the site once, for the putter's player and the relay

Before, the member passing a page link on downloaded it twice, independently:
their mpv straight from the CDN and the relay's pulls again, with a
different format pick and without yt-dlp's cookies. The two drifted (the
putter played on while viewers stalled, or the reverse) and doubled the
connections the CDN saw. Now one SharedSource per track reads the site
once (resuming after cuts, backing off on 412/429/5xx and silence) into a
bounded buffer (64 MiB ahead, 16 behind) that serves both the putter's
mpv through a loopback endpoint and the server's pulls; the putter plays
the relay's own format pick, and the relay sends yt-dlp's cookies.
Up next's warm relay uses the same reader. Pieces tracks (HLS/DASH) keep
two fetches for now. Measured in the opt-in e2e: half the CDN traffic, no
412s under a lockout CDN, viewer seeks 0.29 s instead of 4 s there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
foxyzandClaude Opus 5.5 committed 2026-10-18 02:05:03 +03:00
1 parent 7715dc479a
commit ec0849eb01
19 files changed
+1894 -101

No files matched your search

+78 -9
View File
@@ -638,8 +638,9 @@ ready.
can, so this is Android's plug-in point).
- Resolve: a Jellyfin item through this app's account (or the plan
resolved ahead for Up next, `warmJellyfin`), else the server's login,
else the member passing it on, else blocked (`openJellyfin`); a web
page through the app's own yt-dlp (`LinkResolver.probe`: yt-dlp's
else the member passing it on, else blocked (`openJellyfin`); a link
this member passes on through its shared reader's loopback URL ("Read
once" under "Relay"); a web page through the app's own yt-dlp (`LinkResolver.probe`: yt-dlp's
stream URLs, an EDL of video and audio, the CDN's headers with its
cookies, the subtitles; see "yt-dlp"), failing in plain words with the
cookies that were used (`ResolveFailure`, D11); a relay not fed yet is
@@ -681,6 +682,10 @@ ready.
- Being a relay source (`RelayFeeder`, `session/relay_feeder.dart`): the
current media's channel and Up next's warm relay (see "Relay"), with
the step-1 re-dial (a source rides through this app's own drop). A
link it passes on is read once for its player and the relay
(`SharedLinks`, `session/shared_links.dart`; "Read once" under
"Relay"): the swap check's reader, else Up next's warm one, else a new
one; it lives while the media does (a sit-out keeps it for the relay). A
Jellyfin item this member put up is passed on only once someone
present needs it (`jellyfinRelayNeeded`: the server's login does not
cover it and a present member does not play that server with its own
@@ -840,8 +845,8 @@ and URL, played together by viewers as an mpv EDL
`relayViewerUrl` / `mpvEdl`). media_kit opens an `edl://` URL unchanged
(it writes it to a playlist and `loadlist`s it) and mpv keeps both
streams in sync across seeks; one URL keeps the app's "one media URL"
path (sync, swap, reopen) as is. The source member plays the page
itself (its own resolve, see "yt-dlp") like any other link.
path (sync, swap, reopen) as is. The source member's own player reads
the same tracks through its shared reader ("Read once" below).
A page with only segmented streams (HLS `m3u8_native`, DASH
`http_dash_segments`, e.g. X's HLS, generic `.m3u8`/`.mpd`) is relayed as
@@ -873,6 +878,64 @@ How bytes move:
this machine). Viewers only ever talk HTTP to the server, so the source
never sees mpv's request pattern. The tracks of a media share the
channel and its flow control (they compete for the same uplink).
- **Read once** (a link passed on: `SharedSource`,
`packages/server/lib/src/shared_source.dart`, kept by the app's
`SharedLinks`): the source reads each track from the site once. Per
track a `RelayBuffer` on the source (64 MiB ahead and 16 MiB behind
for all tracks, split by size like the server's, plus the pins:
`sharedAheadBytes` / `sharedBackBytes`), filled by one sequential
upstream reader, serves two readers: the source's own player, through
a loopback endpoint (`http://127.0.0.1:<port>/<random secret>/<track>`,
an mpv EDL for several tracks, no headers: the site's stay in the
reader) answered with the server's own byte-answer code
(`writeRelayRange`, below: headers within 4 s, `connection: close`),
and the server's pulls (`SharedSource.upload`, served by the usual
`RelaySource`). The player opens at once; the endpoint waits for the
tracks' sizes. The two readers are kept within one window
(`RelayBuffer.together`): while both have a request open, the one more
than the track's `ahead` past the other waits without asking upstream,
so one sequential read serves both instead of handing over at every
slice. The server's read-ahead thus stops about 64 MiB past the
source's player, and a source whose upload lags holds its own player
there (viewers would wait at that point anyway); either reader alone
reads freely, pinned slices always come. A seek (shared, so both
readers go together) is a new upstream read there, as on the server.
Upstream (`ResumingBytes`): a cut resumes from the byte it reached
after 100 ms; an answer without a byte (412, 429, 5xx, a refused or
dead connection, nothing for 20 s) backs off 0.5 s doubling to 8 s;
60 s without a byte fails the read, and the buffer's own backoff asks
again. 403/410/412 first resolve the page again (see "yt-dlp"). The
source's player plays the relay's tracks, so the source and the
viewers watch one format, and the reader sends the headers and cookies
yt-dlp gave for it. The putter's swap check is this reader (its sizes
are the media's: no second read); an Up next link passed on feeds its
warm relay from its own reader, the preload player reads its loopback
URL (one prepared on the site before the reader existed is prepared
again), and the swap carries both on. Reload video opens the loopback
URL again (the reader resolves an expired page itself). Not read once:
a page with a pieces track (HLS/DASH, X's): its player plays yt-dlp's
pick from the site and the relay reads the pieces itself, two fetches;
a file passed on (disk) and a Jellyfin item (see "Media pipeline")
keep their own paths. Measured (server `single_fetch_e2e`, opt-in: a
real server and `RelaySource`, mpv 0.41 with the app's pins on
loopback, a fake CDN with Bilibili's Referer rule and two tracks:
60 s of 720p h264 at 4 Mbit/s and 128 kbit/s aac; 20 s of play, then
a seek to 45 s and back to 5 s):
| CDN / source | First frame: source, viewer | Viewer stalls | Viewer seek to 45 s | CDN requests (player + relay) | CDN MiB | 412s |
|---|---|---|---|---|---|---|
| clean, two fetches | 0.22 s, 0.46 s | 0 | 0.27 s | 8 (2 + 6) | 51.8 | 0 |
| clean, read once | 0.27 s, 0.33 s | 0 | 0.29 s | 6 (0 + 6) | 25.9 | 0 |
| cuts every 0 to 2 MiB, two fetches | 0.22 s, 0.38 s | 1 | 0.27 s | 64 (28 + 36) | 55.9 | 0 |
| cuts, read once (29 resumes) | 0.22 s, 0.27 s | 0 | 0.27 s | 33 (0 + 33) | 25.9 | 0 |
| cuts + 412 lockout (> 40 requests in 10 s), two fetches | 0.21 s, 0.39 s | 1 | 4.03 s | 49 (23 + 26) | 35.3 | 6 |
| cuts + lockout, read once | 0.26 s, 0.33 s | 0 | 0.29 s | 33 (0 + 33) | 25.9 | 0 |
| 768 KiB/s per connection, two fetches | 0.40 s, 0.59 s | 1 | 0.80 s | 12 (5 + 7) | 34.0 | 0 |
| 768 KiB/s, read once | 0.40 s, 0.47 s | 1 | 0.76 s | 7 (0 + 7) | 16.9 | 0 |
The source's own seeks took about what the viewer's did; back to 5 s
took 0.25 to 0.29 s everywhere. The source's player stalled only on
the slow CDN (once, in both modes).
- Upstream GETs (`HttpRelayBytes.read`) are split into back-to-back
ranges of at most yt-dlp's `http_chunk_size` for the format, else
10 MiB (`relayUpstreamChunkBytes`); a buffer pull stays one read.
@@ -918,7 +981,9 @@ How bytes move:
source's upstream dropping, a CDN refusing) is followed by the next
one after 250 ms, doubling up to 8 s (`relayRetryDelay`), back to none
once a slice or a pull completes: a failing CDN is not asked again
every round trip (that drew Bilibili's 412/403). A viewer reads the
every round trip (that drew Bilibili's 412/403). Read-ahead never
fetches the slice that would be evicted next, and a long pull evicts
as its slices come, so a buffer stays within its budget. A viewer reads the
slice being filled as its bytes arrive, not once it is whole (mpv's
demuxer, blocked in a read, takes a seek only once that read returns).
Nothing is fetched while nobody watches through the relay (a Jellyfin
@@ -1126,7 +1191,8 @@ all three link modes. Nothing is bundled or installed by hand.
format's `http_headers` plus its `cookies` as a Cookie header; the
requested subtitles but Bilibili's danmaku, an XML comment feed); and
what a relay reads (`pickTracks`), each format with its URL,
`http_headers` and `http_chunk_size`. For a relay only plain http(s)
`http_headers` plus its `cookies` as a Cookie header (what a player
sends) and `http_chunk_size`. For a relay only plain http(s)
formats without DRM count:
- the best single file (`pickSingleFile`: video and audio not `none`, a
video container, tallest up to 1080p, else the shortest above), unless
@@ -1176,11 +1242,14 @@ all three link modes. Nothing is bundled or installed by hand.
track's playlist must be readable, else the swap is refused ("Live
streams can't be passed on", DRM, or nothing a relay can read) with a
hint to use "Everyone fetches it". The swap names the tracks; its relay
source reads them with yt-dlp's headers (`PageRelayMedia`). On
source reads them with yt-dlp's headers and cookies (`PageRelayMedia`),
and the putter's own player plays those same tracks through the one
reader ("Read once" under "Relay"), not yt-dlp's `requested_formats`:
the putter sees the relay's pick (HEVC first) like the viewers. On
403/410/412 (expired format URLs, 6 h on YouTube; signed segment URLs;
Bilibili's 412) from any track it resolves the page again once for all
tracks (not again within 60 s: a further refusal fails the read and the
server's backoff asks later), finds the same
tracks (not again within 60 s: a further refusal fails the read and is
backed off), finds the same
format ids with the same sizes and piece counts and roles (playlists
re-read and re-mapped by index; else "the video changed on the page")
and goes on from where it stopped. Format URLs are bound to the
+11 -3
View File
@@ -94,7 +94,10 @@ Three suites, run from each package's folder:
- `cd packages/server && dart test` (well under a minute): live servers
on loopback: auth and version refusal, identity, takeover and grace
(`presence_test` on a fake clock), media and relay byte serving with
source drops, Jellyfin relay against a fake, queue (and the shared
source drops, the read-once source (`shared_source_test`: its loopback
range answers, a cut and resume with backoff, seeks of either reader,
the two kept within one window, its memory bound), Jellyfin relay
against a fake, queue (and the shared
countdown on a fake clock), session log,
isolate `serve`, sharing (`share_test`: a listener attached and detached
under the same members and Up next, the host key standing in for the
@@ -114,7 +117,10 @@ Three suites, run from each package's folder:
Opt-in, skipped unless set: `JELLYFIN_TEST_*` (core `jellyfin_live` and
server `jellyfin_relay_live` against a real Jellyfin; read their headers),
`YTDLP_LIVE=1` (downloads yt-dlp and resolves a real tweet). Everything
`YTDLP_LIVE=1` (downloads yt-dlp and resolves a real tweet),
`WT_SINGLE_FETCH=1` (server `single_fetch_e2e`: a link passed on read
once against twice, two real mpv against a fake CDN that cuts and locks
out; minutes; `WT_SF_SCENARIOS`, `WT_SF_PLAY` narrow it). Everything
else is offline: Jellyfin and link hosts are local fakes.
Real-mpv tests: `sync_playback_e2e`, `coord_presence` (drops through a
@@ -129,7 +135,9 @@ stand-in yt-dlp and a Referer-checking CDN: the EDL, the headers and
cookies, subtitles without danmaku, a 403, the D11 error),
`media_switch_e2e` (commands during a switch wait and apply once, no mpv
errors), `relay_e2e` (a dedicated server with no file of its
own: a guest's file, links passed on or fetched by the server, a late
own: a guest's file, links passed on (read once: the source's player on
its loopback reader, nothing from mpv at the origin) or fetched by the
server, a late
source, a source that drops and resumes; a seek past what a slow source
has sent, timed; a source that drops while a viewer seeks past
the buffer and another joins: no seek loop, "Waiting for", all resume
+11
View File
@@ -170,6 +170,17 @@ seek or a new link; the start of a freshly put-up link was never
measured on its own. Levers if it still shows: start the window from
the last rate, prefetch from the source as soon as a link goes up,
a smaller gate buffer when only the relay is behind.
Left of the read-once source (ARCHITECTURE "Relay", "Read once"): a
human check against real Bilibili and YouTube (the 412 storm gone, the
putter's picture the relay's format, a Windows putter's loopback), and
the source's first frame and seeks over a 12 Mbit/s uplink
(`tool/shaper`): a putter whose upload lags now holds its own player
about 64 MiB past it. Not built: pieces tracks (HLS/DASH, X's) are
still fetched twice (the putter's player from the site, the relay's
pieces on their own; the same idea needs a `PieceBuffer` and a loopback
playlist); Jellyfin items a member passes on are still streamed twice
when the relay is needed. The e2e's media (26 MiB) fits the putter's
buffer whole, so the 64 + 16 MiB bound is only unit-tested.
Left of step 4 (library): a human check of the Files library on the
real folder (`/mnt/torrents/Torrents/Binux/`: Bleach over three folders
with 66–107 missing, SAO S3 Alicization with OP/ED, Re.Zero S04E17, the