Protocol 12: Up next (a shared queue) and where the video comes from

Up next: a server-side queue anyone edits by item id (add at end or
next, move, remove, play now, skip); each item says who added it and
how it plays. At the real end the first item swaps in after the end
card's countdown, put up by whoever can (its adder for through-me links
and files, someone with access for Jellyfin, else anyone); items that
can't play are skipped with the reason. The next item's link is
resolved ahead. Greyed "Continues with" episodes below. A chip with the
controls opens it as a popover (leaf_widgets, local path dependency for
now); "Add to queue" next to Put it up everywhere.

Delivery: the server measures the source's rate, the media's bitrate
and each member's read rate; the members card shows a source line
(who passes it on, through what, keeping up or not, source left) and
each member's route and buffer. A slow source is told once per media.
A failed server fetch is announced to everyone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
foxyzandClaude Opus 5.5 committed 2026-10-11 06:40:26 +03:00
1 parent 95cd668cfb
commit 6b10cae4e6
50 files changed
+4239 -157

No files matched your search

+110 -6
View File
@@ -571,7 +571,28 @@ How bytes move:
token after a resume); a different size (or piece count and roles)
drops that track's buffer.
`state.media.relay.live` follows the channel, and viewers show "waiting
for <name> to pass the video on" while it is down.
for <name> to pass the video on" while it is down, "<name> left:
playing what the server has" once the source left the server.
- Stats (`relay_stats.dart`, `ServerHub.sampleRates` on the 1 s
heartbeat; wire in [PROTOCOL.md](PROTOCOL.md)): the source's rate is
its bytes over the time a pull is in flight (`FeedClock`, one per media
shared by its tracks, run by each buffer's pull), so a full buffer with
an idle source does not read as a slow one; smoothed over about 4 s of
that time (`SourceRate`). Its first-byte wait counts, so it reads a bit
under the uplink on short pulls. The need is the tracks' sizes over the
reported duration (pieces: their bytes per second once a minute of
segments is in). Each member's rate is its `bytesServed` change over
5 s (`ReadRates`), which follows what its player asks for, not what its
link could take. Two significant digits keep `state` from changing with
every byte. The app (`session/delivery.dart`) derives the rest from the
state: the kind (link, host's file, server's file, member relay, server
fetch, Jellyfin), each member's route (source; via server: relayed,
`/m`, or Jellyfin `jf` false; direct: links, Jellyfin `jf` true) and
buffer health (enough for `bufferNeed`, low, empty while playing). The
members card (`shell/delivery_line.dart`) shows it as a source line and
a dot per member; nothing goes on the picture. The source gets one toast
per media while `rate` < `need`. Not shown: a direct member's speed
(the server never sees its bytes).
Measured (mpv 0.41 viewers, real server and source, a shaped link
between source and server; `tool/relay_e2e.dart`, 2-minute 1080p test
@@ -727,8 +748,8 @@ Client side (`lib/src/jellyfin/session_media.dart`, `progress.dart`,
here for <server>, and nobody is passing it on" with a button that
opens Integrations over the session; linking (or reconnecting) the
account retries by itself. A 401 marks the account expired. The member
list shows "via host" for members whose `jf` is false for the current
item. A reconnect to the same item with `OwnPlay` keeps playing as is.
list shows "via server" for members whose `jf` is false for the current
item (see "Relay", Stats). A reconnect to the same item with `OwnPlay` keeps playing as is.
- **Progress** (`JellyfinProgressReporter`, `OwnPlay` only): Playing once the
file is open and mpv has picked its tracks; Progress every 10 s and at
once on pause, resume, a seek (position more than 3 s from where playback
@@ -758,7 +779,9 @@ Client side (`lib/src/jellyfin/session_media.dart`, `progress.dart`,
`state`; a late joiner never saw that swap), so clients agree without a
wire change. Cancel is local: it stops the countdown on that screen, and
the swapper's Cancel stops the swap; a seek by anyone ends `finished`
and with it every countdown.
and with it every countdown. Only while Up next is empty and its
continuation was not cleared (see "Up next"); a queued item replaces
a countdown already running.
- **Tracks**: the controls' Audio and Subtitles buttons list
`PlaybackController.tracks`. A pick is applied, reported, and saved as a `TrackChoice`
(language, title, codec, channels, forced/HI: from the Jellyfin streams
@@ -771,6 +794,86 @@ Client side (`lib/src/jellyfin/session_media.dart`, `progress.dart`,
Link goes through `mediaForLink`, a Jellyfin pick as a ref, a file
through `ServerIsolate.setFile` (host) or the relay (`swapFile`).
## Up next
The queue: what plays after the current media. The server holds the list
(`EngineQueue` in `packages/core/lib/src/queue.dart`, in memory, part of
`state`; wire in [PROTOCOL.md](PROTOCOL.md), "Queue"), any member edits
it by item id (ops never clash), items outlive their adder. The app side
is `lib/src/queue/`: `QueueController` (owned by the coordinator, fed by
every `state` and feed), the pure rules in `queue_logic.dart`, the UI in
`queue_ui.dart`.
- **Adding** (`QueueController.add`, from "Add to queue" anywhere or the
popover): a link is probed like a swap (`AppYtDlp.probe`) and refused
when it will not play, in the chosen mode ("Through me" also needs
something a relay can read, "Server fetches it" a password on the
server; alone it is direct). The item carries the title, a second line
(uploader or site, the show and episode, a file's parsed episode), the
length when known, and a picture (a page's thumbnail, a Jellyfin
episode still or backdrop, a file's frame from `thumbnailFor`; JPEG,
under the poster cap) sent as `thumb`. A link passed on "through me"
and a file go up as a `key` only: the app keeps the URL or path in
memory under it and nothing else leaves it. The adder gets a toast
"Added … to Up next"; everyone else an activity-log line.
- **Who puts an item up** (`queuePerformer`, from `state` alone, so all
clients agree): a link passed on or a file, only its adder (gone:
skipped, "Bob left"); a Jellyfin item, its adder when it plays that
server with its own account, else the lowest member id that does
(`jf`, which lists every working account; nobody: skipped); a link
everyone opens or the server fetches, its adder when present, else the
lowest active member id; for these, stale or sitting out counts as
away (an adder who is stale is waited for until it is gone). A skip is
a `remove` with the reason from the lowest member id that is not stale
(`queueJanitor`); the feed's `queue-skip` toasts on every screen.
- **Advancing**: at `finished` with Up next waiting, every screen shows
`UpNextCard` (picture, title, who added it and how; with auto-advance on
a 10 s countdown with Cancel, else Play now only); at zero the app that
puts the first item up does, with a swap naming it (`swap.queueId`), so
the server takes it off in the same step and a second swap of it is
ignored. Cancel is per screen like the next-episode countdown. "Play
now" anywhere (a row, the card, the chip's ▶, `>` / Shift+N) swaps
directly from the app that can, else sends `play` and that app reacts
to `state.queuePlay`. With nothing up at all, the first item goes up as
soon as it is added. The swap goes through the usual paths
(`swapPicked`, `swapJellyfin`; a server-fetched item through
`swapQueued`, without the URL, its picture as the art), so relays, art
and refusals work as for any swap; when it fails the item is skipped
with the reason. A swap without `queueId` (a manual one) leaves the
queue alone. A member who puts its own link or file up while sitting
out comes back first; any other queued swap leaves the swapper out.
- **Getting ready** (`queueWarmDue`): in the last 90 s of the current
media, once it is buffered to its end, or at `finished`, the app that
will put the first item up resolves its link (`AppYtDlp.probe`, kept 10
minutes, which the swap then reads), so the swap does not wait for
yt-dlp. A Jellyfin item, a file or a server-fetched link has nothing to
get ready there. Only the first item, only that app, nothing on the
server. Its row shows a dot: hollow waiting, pulsing getting ready,
filled ready, red failed (tried again when it comes up).
- **Continues with**: not queued, computed here like the next episode:
with an own-account Jellyfin episode playing, the next three episodes
of the series (`/Shows/{id}/Episodes`, cached), greyed under Up next,
"+" adds one. "Clear" hides them for the media playing and keeps its
next-episode countdown from starting.
- **UI**: `UpNextChip` top right under the top bar (under the party in
fullscreen), shown with the controls like the pills (`PillGate`) and
while its popover is open: "Up next · title", a +n count, ▶ plays it
now; without Up next the continuation ("Next · S1E8 …", ▶ = next
episode); nothing when both are empty. Its title opens the queue as a
`LeafPopover` (package `leaf_widgets`, an overlay rather than a route):
frosted glass like the app menus; rows with a 16:9 picture, title, second
line, "length · added by (avatar) name · how", a drag handle
(`ReorderableListView`, reordered here at once and sent as `move`), ▶
and × floating in on hover; "Continues with"; a "Paste a link to add"
field (focused on open, so the player's keys and Space rest while it
is up; Esc closes) and a drop zone for links and video files. Being no
route, it holds the shell's own drop and paste handling off while open
(`handoffHolds`). "Add to queue" elsewhere: the Link page's button
next to Put it up (in the chosen mode), a right-click menu on recent
links, Jellyfin cards, episodes and details-panel episodes, and Files
videos (Put it up / Add to queue / Play next), and a button in the
Jellyfin details panel (right-click: play next). None of it offline.
## Desktop UI
One window, `AppShell` (`lib/src/shell/`), whatever the connection state
@@ -802,8 +905,9 @@ from `AppMenuStyle.theme`; the uosc controls keep their own flat menus.
dialog; no avatars or chat), online the party: `AvatarStack` (host first, then
ready; spreads on hover; 5 shown, 3 under 1100 px), the server name and
"ready n/m" (hidden under 1100 px), chat with an unread badge, and Leave
(asks first). The stack opens the members card (status, host / relay
pills, "via host", "sitting out"; per member force ready, Nudge and Sit
(asks first). The stack opens the members card (the source line, then
per member status, route and buffer dot (see "Relay", Stats), host /
relay pills, "sitting out"; per member force ready, Nudge and Sit
out (anyone, the host too), your own row Sit out / Join back; "Copy invite
address": the dialled address when joined; hosting, `inviteAddresses`
of `NetworkInterface.list` (IPv4, no loopback or link-local, private
+87 -12
View File
@@ -1,4 +1,4 @@
# Protocol (proto = 11)
# Protocol (proto = 12)
Wire format only. Behavior (gate, clock, timeouts) is in
[ARCHITECTURE.md](ARCHITECTURE.md). Source of truth:
@@ -30,7 +30,7 @@ with different `proto` refuse each other at `hello` (`proto mismatch`).
## Client to server
```
{"t":"hello","proto":11,"secret":"","user":{"name":"","avatar":""},
{"t":"hello","proto":12,"secret":"","user":{"name":"","avatar":""},
"media":{"name":"","size":0,"duration":0,"url":null},"hostKey":"...",
"resume":"<token>"}
{"t":"ready","ready":true,"source":"manual"} source: manual | auto
@@ -39,6 +39,7 @@ with different `proto` refuse each other at `hello` (`proto mismatch`).
{"t":"swap","media":{"name":"","size":0,"duration":0,"url":"..."}}
{"t":"swap","media":{"name":"","size":0,"duration":0,"jellyfin":{<ref>}}}
{"t":"swap","media":{"name":"","size":123,"duration":0,"url":"...","relay":{}}}
{"t":"swap","media":{...},"queueId":"q3"} puts up an Up next item
{"t":"report","position":123.1,"buffered":6.2,"eventId":41,
"duration":1440.5,"mediaId":2,"jf":{"<serverId>":true},"error":"Not allowed (403)"}
{"t":"ping","c":1767225600000}
@@ -47,6 +48,7 @@ with different `proto` refuse each other at `hello` (`proto mismatch`).
{"t":"nudge","target":"<memberId>"}
{"t":"profile","name":"...","avatar":"<base64>"}
{"t":"art","mediaId":2,"poster":"<base64>","banner":"<base64>"}
{"t":"queue","op":"add"|"move"|"remove"|"play"|"skip",...} see "Queue"
```
Field notes:
@@ -74,7 +76,10 @@ Field notes:
- `report.jf` (optional): Jellyfin access per `ServerId`, `true` = this
client plays the item with its own account, `false` = through the relay.
Sticky: a report without `jf` keeps the last one. At most 8 entries, keys
up to 64 characters; others are dropped.
up to 64 characters; others are dropped. The reference client lists
every server it has a working account on as `true`, then the current
item's server as it plays it (so a queued Jellyfin item finds someone
to put it up while other media plays).
- `report.error` (optional, omitted when empty): short plain text saying
why this client's player cannot open the media `report.mediaId` (e.g.
`Not allowed (403)`). The client sends it in every report while that
@@ -127,8 +132,8 @@ Field notes:
"serverTime":"<ISO-8601 UTC>","mediaId":1,
"media":{name,size,duration,url?,jellyfin?,relay?,swapBy?,art?}|absent,
"members":[{"id","name","avatarHash"?,"ready","inSet","buffered","stale","jf"?,
"host"?,"out"?,"error"?}],
"seekBy":"m1","seekSeq":3,"relay":true}
"host"?,"out"?,"error"?,"rate"?}],
"seekBy":"m1","seekSeq":3,"relay":true,"queue":[<item>]?,"queuePlay":"q3"?}
{"t":"pong","c":1767225600000,"s":1767225600012}
{"t":"feed","actor":"","action":"","target":"","reason":"","at":"<ISO-8601>"}
{"t":"chat","text":"...","from":"<name>"}
@@ -161,17 +166,23 @@ Field notes:
server's `hostKey` (see `hello.hostKey`). `members[].out` (omitted when
false): the member sits out (see `out`). `members[].error` (omitted when
empty): the member's player cannot open the current media (see
`report.error`).
`report.error`). `members[].rate` (omitted when 0): bytes per second
the server sent this member over the last 5 s (`/m`, `/j` and `/r`
together), two significant digits, under 10 kB/s counts as 0. Members
that fetch the media themselves read nothing from the server.
- `pong.c` echoes the ping's `c`; `pong.s` is the server's clock in epoch ms
(the clock `serverTime` uses). With `t3` the local receive time, the offset
is `s - (c + t3) / 2`, accurate to half the round trip.
- `feed.action` values: `ready`, `unready`, `force-ready`, `force-unready`,
`force-play`, `force-hold`, `out`, `back`, `join`, `leave`, `stale`, `swap`,
`seek`, `pause`, `resume`, `finished`, `chat`, `rename`, `avatar`. `target`
`seek`, `pause`, `resume`, `finished`, `chat`, `rename`, `avatar`,
`queue-add`, `queue-remove`, `queue-skip`, `fetch-fail` (proto 12: the
server could not fetch the current media, `target` its name). `target`
and `reason` are optional; `reason` carries `auto`/`manual` for ready and
`silent too long` or `no signal` for leave/stale. `rename`: `actor` is the
new name, `target` the old one. `avatar`: the actor changed only the
picture.
picture. `queue-*`: `target` is the item's title, `reason` (skip) why it
could not play (see "Queue").
- Chat is delivered twice: a `chat` frame with the text and a textless
`feed` with `action:"chat"`. Render the `chat` frame only; the reference
client drops the textless feed entry.
@@ -207,8 +218,9 @@ state media.art = {"poster":"<hash>","banner":"<hash>"}
- `media.art` carries hashes like `avatarHash`; a key is omitted when that
picture is absent, `art` when both are. The same picture as poster and
banner (one hash) is a web video's thumbnail.
- `GET /i/<sid>/<token>/<hash>`: as `/a/` (see "Avatars"); 404 when the
current media has no such art.
- `GET /i/<sid>/<token>/<hash>`: as `/a/` (see "Avatars"); also the
pictures of queued items (`queue[].thumb`). 404 when neither the current
media nor a queued item has it.
## Jellyfin media
@@ -257,6 +269,7 @@ swap media.relay = {} the swapper's machine serves the bytes
media.relay = {"fetch":true} the server reads media.url itself
state media.relay = {"source":"m2","live":true} or {"fetch":true,"live":true}
media.relay = {"fetch":true,"live":true,"tracks":[{"kind":"video"},{"kind":"audio"}]}
media.relay = {"source":"m2","live":true,"rate":4100000,"need":2300000}
```
Tracks: a relayed media is one file, or tracks that viewers play as one
@@ -286,6 +299,15 @@ anything else means one file.
channel is connected, or the server could read the link). `tracks`: as
the swap named them, or, with `fetch`, set once the server has read the
link (together with `live`).
- `state.media.relay.rate` (omitted while unknown): bytes per second the
source (with `fetch`, the server's own fetch) delivers while the buffer
is waiting on it, smoothed over about 4 s of that time and present once
2 s of it are measured; it holds while the buffer is full and the
source idles. `need` (omitted while unknown): the media's average bytes
per second, the tracks' sizes over `media.duration` (a pieces track:
its segments in so far, after a minute of them). Both two significant
digits, updated once a second. `rate` under `need` means the source
cannot keep up; clients tell the source once per media.
- `welcome.mediaUrl` for relayed media without `jellyfin`: this member's
`/r/` URL; absent for the source itself (it plays its own copy) and for
a fetched link the server has not read yet (clients open it on `live`).
@@ -360,6 +382,59 @@ is a fresh `hello` (with `resume` = the old token), after which the client
re-sends its prior ready flag. If the old member was still there (unclean
drop), `resume` takes it over and the token and media URL stay the same.
## Reserved, not implemented
## Queue (proto 12)
Server-held queue (`queue-add`, `queue-skip`) would auto-advance like a swap.
Up next: what plays after the current media, held by the server for
everyone, in memory (a restart loses it). Any member edits it; items stay
when their adder leaves. Behavior (who puts an item up, the countdown,
skipping) is in [ARCHITECTURE.md](ARCHITECTURE.md), "Up next".
```
client {"t":"queue","op":"add","at":"end"|"next","item":{<item>},"thumb":"<base64>"}
{"t":"queue","op":"move","id":"q3","to":0}
{"t":"queue","op":"remove","id":"q3","why":"Bob left"}
{"t":"queue","op":"play","id":"q3"}
{"t":"queue","op":"skip"}
{"t":"swap","media":{...},"queueId":"q3"}
state "queue":[<item>,...], "queuePlay":"q3"
<item> = {"id":"q3","media":{<swap media>},"how":"direct"|"server"|"through"|"file"|"jellyfin",
"title":"...","sub":"...","duration":213,"thumb":"<hash>",
"by":"m2","byName":"Bob","key":"<adder's handle>"}
```
- `add`: `item` without `id`, `by`, `byName`, `thumb` (the server sets
them: `id` `q<n>`, unique for the server's life; who sent it and their
name then). `at` `next` puts it first, else last. `thumb` (optional):
base64 picture, checked and capped like a poster (256 KiB decoded);
`item.thumb` is its hash, served at `/i/` (see "Media art") while the
item is queued. `title` and `sub` are cleaned like media names and cut
to 200 characters (an empty title becomes the media's name); `duration`
(seconds, omitted when unknown) is informational. At most 100 items;
over that the sender gets `err` `Up next is full (100 items)`.
- `how` and what `media` must carry:
- `direct`: an http(s) `url`; everyone opens the link.
- `server`: an http(s) `url` and `relay` `{"fetch":true}`, on a server
with a password (as for a fetch swap). The server keeps the URL to
itself: the broadcast `media` has none.
- `through` (a link the adder passes on) / `file` (the adder's file):
`name` and `size` only, and `key` (8 to 64 of `[A-Za-z0-9_-]`): the
adder's own handle for the link or path, which never leave its app.
A `url` sent along is dropped.
- `jellyfin`: a `jellyfin` ref, checked like a swap's (any `media` with
a ref is `jellyfin`).
Anything else: `err` `bad message: ...`.
- `move`: `id` to index `to` (clamped). `remove`: drops `id`; with `why`
(cut to 120 characters) it is a skip (feed `queue-skip` with the
reason; clients show it), else feed `queue-remove`. `add` gives feed
`queue-add`. Unknown ids are ignored.
- `play`: moves `id` first and sets `state.queuePlay` to it: a request for
the member who can put it up to do so now. `skip` is `play` of the
first item. `queuePlay` ends with any swap, when the item is removed,
or after 30 s.
- `swap.queueId`: the swap puts that item up and takes it off the queue
in the same step. A swap naming an item that is no longer queued is
ignored (someone put it up first). For a `server` item the swap may
leave out `url`: the server uses the one it kept. A swap without
`queueId` leaves the queue as it is.
- `state.queue` (omitted when empty) and `state.queuePlay` (omitted when
none) ride in every state, `welcome.state` included.
+16 -11
View File
@@ -12,7 +12,8 @@ off by deleting it in the same commit as the code. Wire changes also edit
ever be skipped.
2. **Files**: next episode from Files (by parsed episode, with the
countdown). Later: tracking for file series.
countdown, and under Up next's "Continues with"). Later: tracking for
file series.
3. **yt-dlp follow-ups** (see ARCHITECTURE "yt-dlp"): check that the
bundled Windows libmpv has the ytdl hook (Lua) and plays `edl://` URLs
@@ -36,7 +37,10 @@ off by deleting it in the same commit as the code. Wire changes also edit
filter on the Schedule from the user's AniList lists. This is the
Schedule's personal tracking; AnimeFlow has no per-user follow.
5. **Ops**: check shaders on Windows; the relay e2e "source drops
5. **Ops**: `leaf_widgets` (the Up next popover's `LeafPopover`) is a
local path dependency: make it a git dependency like `fox_utils`
before the next release (CI cannot see the path), or go back to the
app's popover route. Check shaders on Windows; the relay e2e "source drops
uncleanly and resumes" fails about half the time under the full suite.
6. **Windows: first real use.** CI builds and publishes
@@ -52,8 +56,6 @@ off by deleting it in the same commit as the code. Wire changes also edit
over instead of the buffer running dry.
- "Bob can't open this link — relay it?": offer the relay to whoever put
a direct link up when a viewer's player fails to open it.
- Show who is relaying and how fast in the member list; warn the source
when its upload is below the media's bitrate.
8. **Jellyfin** (epic, below).
@@ -77,19 +79,22 @@ off by deleting it in the same commit as the code. Wire changes also edit
- HTTP: move the app's clients (Jellyfin, AniList, AnimeFlow, updater)
to dio from `fox_utils` when touching them, for retries/interceptors.
- TLS: document a reverse proxy in front of the server.
- Playlist: `queue-add`, `queue-skip`, auto-advance like a swap.
- Up next follow-ups (see ARCHITECTURE "Up next"): getting the first
item ready only resolves its link; fetching its first seconds early
through the relay would make relayed starts instant too. A member
whose reconnect lands as a new member loses its "through me" links
and files in the queue (skipped as gone); the key could follow the
member. A whole season or a pick list at once ("Add the rest of the
season").
- Sidecar subtitles.
## Epic: Jellyfin
What is left (the rest is in [ARCHITECTURE.md](ARCHITECTURE.md), "Jellyfin
in a session"):
- Queue: put several library items (a season, a pick list) up as a
playlist that advances like a swap (shares `queue-add` / `queue-skip`
with the backlog's Playlist item).
- Auto-advance cancel for everyone: today Cancel is per screen (the
designated swapper's Cancel stops the swap; a seek stops all). A shared
cancel needs a wire message.
- Auto-advance cancel for everyone (the next episode and Up next): today
Cancel is per screen (the designated swapper's Cancel stops the swap; a
seek stops all). A shared cancel needs a wire message.
- Sidecar (external) subtitle streams of Jellyfin items: only embedded
tracks are mapped to mpv today.