Relay: a member's media through the server's buffer (proto 8)
Viewers who cannot reach the media get it through the server: a file on
any member's computer, a link passed on by a member ("Through me") or
fetched by the server ("Server fetches it", password-protected servers
only), and Jellyfin items for guests without an account, sourced by
whoever put them up with their own sign-in (or a pasted link's key),
which never leaves their machine. Fixes guests on a dedicated server
without a Jellyfin login being blocked.
The source dials /r/.../source (works behind NAT) and answers pulls with
an adaptive in-flight window (2 x rate x min RTT, 256 KiB..8 MiB). The
server keeps a memory-only buffer of 1 MiB slices sized like a viewer's
mpv buffer (pinned head/tail, 64 MiB back, 256 MiB ahead), fills it with
one sequential pull at a time (blocked viewers first, newest request
only), fetches nothing while nobody watches through it, and drops it on
the next swap or stop. Measured with mpv through shaped links
(tool/relay_e2e.dart): 400 Mbit/s seeks ~0.03 s; 40 Mbit/s no stalls,
3 viewers cost the source ~1x the bytes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
foxyzandClaude Opus 5.5 committed
1 parent
9c5733719b
commit
e7f33b5dc3
27 files changed
+2438
-116
No files matched your search
+96
-7
@@ -8,7 +8,7 @@ Plain overview first: [HOW-IT-WORKS.md](HOW-IT-WORKS.md). Wire format:
|
||||
| Path | What | Rule |
|
||||
|---|---|---|
|
||||
| `packages/core` | protocol messages, sync engine (`SyncEngine`), constants, config, Range parsing, logging, Jellyfin client | pure Dart, no Flutter |
|
||||
| `packages/server` | Relic HTTP + WebSocket server (`SyncServer`, `ServerHub`, `Session`, media endpoint, Jellyfin relay), the `serve()` entrypoint, `bin/server.dart` (dedicated server) | pure Dart, no Flutter |
|
||||
| `packages/server` | Relic HTTP + WebSocket server (`SyncServer`, `ServerHub`, `Session`, media endpoint, relay buffer, Jellyfin relay), the `serve()` entrypoint, `bin/server.dart` (dedicated server) | pure Dart, no Flutter |
|
||||
| `apps/desktop` | Flutter app: UI, `SyncClient`, `SessionCoordinator`, `PlaybackController` (media_kit/mpv), `ServerIsolate`, updater | ships with `watch-together-server` |
|
||||
| `tool` | harnesses: `range_logger`, `shaper`, `fake_viewer`, `e2e_check` | dev only |
|
||||
| `deploy` | systemd unit for the dedicated server | |
|
||||
@@ -144,7 +144,8 @@ Additive wire changes only; bump `proto` and PROTOCOL.md in the same commit.
|
||||
|
||||
## Media
|
||||
|
||||
Three sources, same sync.
|
||||
Three sources, same sync. Any of them can also go through the relay
|
||||
(below).
|
||||
|
||||
- **Link**: the media field holds an http(s) URL. It travels as `media.url`
|
||||
in `swap` and `state`, and as `mediaUrl` in `welcome`. Each app plays it
|
||||
@@ -166,8 +167,8 @@ Three sources, same sync.
|
||||
- **Jellyfin item**: the media is a ref (`{serverId, itemId, ...}`, wire in
|
||||
[PROTOCOL.md](PROTOCOL.md) "Jellyfin media"); the server never stores or
|
||||
sends a URL or token for it. Each app resolves it with its own account and
|
||||
direct-plays the static stream; guests use the relay (see "Jellyfin
|
||||
relay").
|
||||
direct-plays the static stream; guests use the server's own login when
|
||||
it has one ("Jellyfin relay"), else the member who put it up ("Relay").
|
||||
|
||||
Range table (`resolveMediaRequest`, tested byte-exact):
|
||||
|
||||
@@ -362,6 +363,86 @@ errors leave it out).
|
||||
subscription does).
|
||||
- No progress reporting in the server: watched state is per client.
|
||||
|
||||
## Relay
|
||||
|
||||
A member's media through the server's memory, for viewers who cannot
|
||||
reach it themselves (`packages/core/lib/src/relay.dart` for the wire and
|
||||
the source, `packages/server/lib/src/relay_buffer.dart`,
|
||||
`relay_routes.dart`; wire in [PROTOCOL.md](PROTOCOL.md) "Relay").
|
||||
|
||||
What gets relayed (`SessionCoordinator.swapPicked`):
|
||||
|
||||
| Put up | Others get it |
|
||||
|---|---|
|
||||
| File, by the host | `/m` from the hosting app (no relay) |
|
||||
| File, by anyone else | relayed from that member's computer |
|
||||
| Link, "Everyone fetches it" (default) | each app fetches it |
|
||||
| Link, "Server fetches it" | the server reads it once into the buffer; the link is never broadcast; refused on an open server (it would fetch any address it can reach for anyone) |
|
||||
| Link, "Through me" | relayed from that member's computer; for links only they can open |
|
||||
| Jellyfin item (browser pick or pasted link) | own account first, then the server's login (`/j`), then relayed from the member who put it up (with their sign-in, or the pasted link's key, which stays on their machine) |
|
||||
|
||||
Relayed links must be one file the relay can range-read: a `text/*`
|
||||
response (YouTube and other pages, which only the player resolves) is
|
||||
refused before the swap.
|
||||
|
||||
How bytes move:
|
||||
- The source dials `/r/.../source` (outbound, so NAT/CGNAT are fine) and
|
||||
answers the server's pulls from the file or the URL (`RelaySource`,
|
||||
`FileRelayBytes`, `HttpRelayBytes`; a Jellyfin source sends its own
|
||||
`Authorization` header from this machine). Viewers only ever talk HTTP
|
||||
to the server, so the source never sees mpv's request pattern.
|
||||
- The server keeps one `RelayBuffer` per relayed media: 1 MiB slices in
|
||||
memory, filled by one sequential pull at a time. Kept: the first and
|
||||
last 2 MiB (container indexes; mpv reads both on open and on seeks),
|
||||
64 MiB behind the slowest viewer and 256 MiB past the furthest, the
|
||||
same as each viewer's mpv buffer (`PlaybackController`), so a skip
|
||||
inside the viewers' buffer window is served from memory. Evicted by
|
||||
distance from the viewers, never LRU. With 2+ viewers the source sends
|
||||
each byte about once.
|
||||
- Filling: a viewer blocked on a slice first (oldest wait first, handing
|
||||
the source over at slice boundaries), then the pinned slices, then
|
||||
read-ahead from the slowest viewer. Nothing is fetched while nobody
|
||||
watches through the relay (a Jellyfin source with only account holders
|
||||
uploads nothing).
|
||||
- mpv leaves superseded requests open but unread after a seek: only a
|
||||
viewer's newest request moves its position or asks the source for
|
||||
anything; older ones still get bytes that are already buffered.
|
||||
- Flow control on the source (`RelayWindow`): one in-flight budget per
|
||||
connection, 2 x delivery rate x min RTT (rate max-filtered over 2 s),
|
||||
clamped 256 KiB to 8 MiB; the server acks bytes as they arrive. A
|
||||
cancelled pull's unacked bytes are written off, and a cancel that lands
|
||||
while a frame is being read stops it before it is sent.
|
||||
- Cleanup: the buffer is memory only. Any swap disposes it (its URLs then
|
||||
404, the source is closed with 4001 and stops), and so does server
|
||||
stop. A crash frees it with the process.
|
||||
- Source away: viewers keep playing from the buffer; a request waits up
|
||||
to 60 s for missing bytes. The source reconnects with backoff (same
|
||||
token after a resume); a different file size drops the buffer.
|
||||
`state.media.relay.live` follows the channel, and viewers show "waiting
|
||||
for <name> to pass the video on" while it is down.
|
||||
|
||||
Measured (mpv 0.41 viewers, real server and source, a shaped link
|
||||
between source and server; `tool/relay_e2e.dart`, 2-minute 1080p test
|
||||
file averaging 10 Mbit/s with a 15 Mbit/s busy section):
|
||||
|
||||
| Source uplink | Open | Seeks (incl. +90 s skip) | Stalls in 50 s |
|
||||
|---|---|---|---|
|
||||
| 400 Mbit/s, 20 ms | 0.8 s | 0.01 to 0.03 s | 0 |
|
||||
| 40 Mbit/s, 40 ms | 2.1 s | 1.1 s first, then 0.03 s | 0 |
|
||||
| 20 Mbit/s, 150 ms | 5.4 s | 1.1 to 5.3 s | 3 |
|
||||
| 12 Mbit/s, 40 ms | 5.4 s | 0.5 to 2.9 s | 3 |
|
||||
|
||||
3 viewers skipping together on 40 Mbit/s: no stalls, the source sent
|
||||
101 MiB for 287 MiB served. A late joiner opened in 0.04 s from the
|
||||
buffer. A 5 s source outage: no stall; 30 s: one stall once the buffer ran
|
||||
out, then it recovered. Freezes of 0.3 to 0.8 s in 30% of seconds: no
|
||||
stall. So the source's upload must beat the media's average bitrate
|
||||
(peaks are covered by read-ahead once it has some lead), and comfortably
|
||||
so for quick seeks. Rejected along the way: a fixed per-request window
|
||||
(seeks of 14 s on slow links, data for abandoned positions clogging the
|
||||
pipe), one shared window across requests (deadlocked on mpv's unread
|
||||
requests), Dart `http2` (connection window stuck at 64 KiB).
|
||||
|
||||
## Jellyfin in a session
|
||||
|
||||
Client side (`lib/src/jellyfin/session_media.dart`, `progress.dart`,
|
||||
@@ -374,8 +455,9 @@ Client side (`lib/src/jellyfin/session_media.dart`, `progress.dart`,
|
||||
MediaBrowser` with this install's fields), not in the URL: `OwnPlay`,
|
||||
`report.jf {serverId: true}`. Else, when `state.relay`, `welcome.relayUrl`
|
||||
or `/j/default/<token>/<itemId>` on the dialled host: `RelayPlay`,
|
||||
`jf: false`. Else `BlockedPlay`: the session shows "This needs a
|
||||
Jellyfin account on <server> (or the host can relay)" with a button that
|
||||
`jf: false`. Else, when `media.relay` names a source, the `/r/` URL:
|
||||
`RelayPlay`. Else `BlockedPlay`: the session shows "No Jellyfin account
|
||||
here for <server>, and nobody is passing it on" with a button that
|
||||
opens Integrations over the session; linking (or reconnecting) the
|
||||
account retries by itself. A 401 marks the account expired. The member
|
||||
list shows "via host" for members whose `jf` is false for the current
|
||||
@@ -577,6 +659,13 @@ this repo, written by `tool/release.sh`.
|
||||
- Plain `ws://`, no TLS: friends use LAN, VPN or a forwarded port; TLS
|
||||
(e.g. a reverse proxy) is a documentation item.
|
||||
- One session per server, id `default`: the map is keyed for later but unused.
|
||||
- No server storage: config and logs only; media never touches the server disk.
|
||||
- No server storage: config and logs only; media never touches the server
|
||||
disk. The relay buffer is memory only and sized like a viewer's buffer
|
||||
(about 320 MiB at most), so a small VPS can carry it.
|
||||
- The relay is the source's own connection through the server, not a
|
||||
shared token: a member's Jellyfin sign-in or a link's key never leaves
|
||||
their machine. Prior art (Plex Relay, Tailscale DERP) relays as a
|
||||
bandwidth-capped fallback on someone else's servers; here the server is
|
||||
the owner's, so it carries full quality.
|
||||
- Avatars ride in the 1 Hz `state` broadcast (cap 256 KiB base64 per avatar):
|
||||
simple, costs host uplink; hash-based serving is on the roadmap.
|
||||
@@ -19,8 +19,11 @@ A five-minute tour. Details: [ARCHITECTURE.md](ARCHITECTURE.md), wire format:
|
||||
video bytes come from somewhere else:
|
||||
- a link (any http/https URL): each app fetches it directly
|
||||
- a Jellyfin item: each app plays it with its own Jellyfin account,
|
||||
friends without one through the server (/j/... endpoint)
|
||||
friends without one through the server
|
||||
- a file on the host's disk: the server streams it (/m/... endpoint)
|
||||
- the relay (/r/...): a member's computer passes a file, a link or a
|
||||
Jellyfin item on to the server, which keeps a few minutes in memory
|
||||
and streams it to everyone who needs it
|
||||
```
|
||||
|
||||
- **App**: the desktop window. It draws the video with mpv, shows who is
|
||||
@@ -31,7 +34,8 @@ A five-minute tour. Details: [ARCHITECTURE.md](ARCHITECTURE.md), wire format:
|
||||
- **Jellyfin** (or any web host): where the video lives. For a Jellyfin
|
||||
item the server only knows which item it is; each app signs in with its
|
||||
own account and reports what it watched. For friends without an account
|
||||
the server fetches the item with the host's sign-in and passes it on.
|
||||
the member who put it up passes it on through the server (their sign-in
|
||||
stays on their computer).
|
||||
|
||||
## Ways to run
|
||||
|
||||
|
||||
+56
-7
@@ -1,4 +1,4 @@
|
||||
# Protocol (proto = 7)
|
||||
# Protocol (proto = 8)
|
||||
|
||||
Wire format only. Behavior (gate, clock, timeouts) is in
|
||||
[ARCHITECTURE.md](ARCHITECTURE.md). Source of truth:
|
||||
@@ -22,7 +22,8 @@ with different `proto` refuse each other at `hello` (`proto mismatch`).
|
||||
`uptimeSecs`; unauthenticated). `GET /m/<sid>/<token>/<name>`
|
||||
serves the host's file (Range table in ARCHITECTURE.md).
|
||||
`GET /j/<sid>/<token>/<itemId>` relays the current Jellyfin media (see
|
||||
"Jellyfin media" below).
|
||||
"Jellyfin media" below). `/r/<sid>/<token>/...` is the relay of a
|
||||
member's media (see "Relay" below).
|
||||
|
||||
## Client to server
|
||||
|
||||
@@ -35,6 +36,7 @@ with different `proto` refuse each other at `hello` (`proto mismatch`).
|
||||
{"t":"seek","position":123.4,"seq":3}
|
||||
{"t":"swap","media":{"name":"","size":0,"duration":0,"url":"..."}}
|
||||
{"t":"swap","media":{"name":"","size":0,"duration":0,"jellyfin":{<ref>}}}
|
||||
{"t":"swap","media":{"name":"","size":123,"duration":0,"url":"...","relay":{}}}
|
||||
{"t":"report","position":123.1,"buffered":6.2,"eventId":41,
|
||||
"duration":1440.5,"mediaId":2,"jf":{"<serverId>":true}}
|
||||
{"t":"ping","c":1767225600000}
|
||||
@@ -71,7 +73,7 @@ Field notes:
|
||||
- `chat.text` is cut to 500 characters.
|
||||
- `swap.media.url` is the link; `name`, `size`, `duration` are informational.
|
||||
`swap.media.jellyfin` makes it Jellyfin media (see below); then `url` is
|
||||
dropped.
|
||||
dropped. `swap.media.relay` makes it relayed media (see "Relay").
|
||||
- Frames before `welcome` (other than `hello`) are ignored; unparsable or
|
||||
unknown frames are ignored silently.
|
||||
|
||||
@@ -104,7 +106,8 @@ Field notes:
|
||||
`playing`, on the server's clock (see `pong`). `finished`: the media
|
||||
reached its end (`position` = duration); it stays until a seek or swap.
|
||||
`eventId` increases on seek, swap and transport changes.
|
||||
`mediaId` increases on swap. `media.duration` is 0 until a client reports
|
||||
`mediaId` increases on swap. `media.relay` is present for relayed media
|
||||
(see "Relay"). `media.duration` is 0 until a client reports
|
||||
it. `seekBy` (member id) and `seekSeq` are present only while the current
|
||||
`eventId` came from a seek. `relay` (omitted when false): the server can
|
||||
relay the current Jellyfin media. `members[].jf` is the member's last
|
||||
@@ -152,6 +155,55 @@ Field notes:
|
||||
credential. Bytes count toward `bytesServed`.
|
||||
- Pasted Jellyfin links sent as plain `url` stay plain links.
|
||||
|
||||
## Relay
|
||||
|
||||
Media whose bytes go through the server's buffer: a file on a member's
|
||||
computer, a link passed on by a member, a link the server fetches itself,
|
||||
or a Jellyfin item for viewers without an account. Behavior (buffer,
|
||||
flow control, cases) is in [ARCHITECTURE.md](ARCHITECTURE.md), "Relay".
|
||||
|
||||
```
|
||||
swap media.relay = {} the swapper's machine serves the bytes
|
||||
media.relay = {"fetch":true} the server reads media.url itself
|
||||
state media.relay = {"source":"m2","live":true} or {"fetch":true,"live":true}
|
||||
```
|
||||
|
||||
- `swap`: `relay` with `jellyfin` keeps the ref (the swapper serves it to
|
||||
viewers without an account; `fetch` is ignored there). `relay` without
|
||||
`jellyfin`: no URL is stored or broadcast; `fetch` needs an http(s)
|
||||
`url` (else `relay fetch needs an http(s) url`), which the server keeps
|
||||
to itself, and a server with a password (else `server fetch needs a
|
||||
server password`: an open server would fetch any address it can reach
|
||||
for anyone). Any later swap ends the relay.
|
||||
- `state.media.relay.source`: member id of the source (absent with
|
||||
`fetch`). `live` (omitted when false): bytes can flow (the source's
|
||||
channel is connected, or the server could read the link).
|
||||
- `welcome.mediaUrl` for relayed media without `jellyfin`: this member's
|
||||
`/r/` URL; absent for the source itself (it plays its own copy). Clients build it after a swap like the `/m` one:
|
||||
`http://<host>/r/default/<token>/<mediaId>/<name>` (name URL-encoded).
|
||||
- `GET /r/<sid>/<token>/<mediaId>/<name>`: the media, Range table as for
|
||||
`/m`. Errors: 401 bad token, 403 kicked, 404 not the current relayed
|
||||
media (a player still on an older `mediaId` gets 404, never the new
|
||||
bytes), 410 unknown session, 503 + `retry-after` the source never
|
||||
connected (60 s). A request whose bytes do not arrive within 60 s ends
|
||||
early. Bytes count toward `bytesServed`.
|
||||
- `GET /r/<sid>/<token>/source` (WebSocket): the source's relay channel,
|
||||
dialled by the source (works behind NAT). Only the current source's
|
||||
token is accepted; anyone else, or the source after the next swap, is
|
||||
closed with code 4001 (the source stops instead of reconnecting).
|
||||
Frames:
|
||||
|
||||
```
|
||||
source -> server {"t":"info","size":<bytes>} first frame
|
||||
binary: [u32 big-endian pull id][bytes] in order per pull
|
||||
{"t":"end","id":1} {"t":"fail","id":1,"reason":"..."}
|
||||
server -> source {"t":"pull","id":1,"start":0,"end":1048575} inclusive
|
||||
{"t":"ack","id":1,"n":65536} bytes arrived
|
||||
{"t":"cancel","id":1} stop that pull
|
||||
```
|
||||
|
||||
A `size` different from the one before (a reconnect) drops the buffer.
|
||||
|
||||
## Join and reconnect
|
||||
|
||||
`hello` then `welcome`, then the app opens `mediaUrl`, seeks to the server
|
||||
@@ -164,6 +216,3 @@ drop), `resume` takes it over and the token and media URL stay the same.
|
||||
## Reserved, not implemented
|
||||
|
||||
Server-held queue (`queue-add`, `queue-skip`) would auto-advance like a swap.
|
||||
`RelayConstants.chunkBytes`, `creditWindowChunks`, `maxConcurrentOpen` and the
|
||||
`open`, `ack`, `cancel`, `eof`, `buffered`, `state`, `event` kinds in
|
||||
`MessageKind` are unused leftovers (the `state` frame does not use the enum).
|
||||
+16
-7
@@ -6,16 +6,25 @@ off by deleting it in the same commit as the code. Wire changes also edit
|
||||
|
||||
## Backlog
|
||||
|
||||
1. **Windows: first real install.** CI builds and publishes
|
||||
1. **Windows: first real use.** CI builds and publishes
|
||||
`watch-together-<ver>-windows-x64.zip` (libmpv bundled) and runs the
|
||||
updater tests on NTFS, but nobody has launched the app on Windows yet.
|
||||
Check: playback, fullscreen controls, hosting (firewall prompt), in-app
|
||||
update from one release to the next. The rest of the desktop tests are
|
||||
not in CI.
|
||||
updater tests on NTFS; a friend has joined from it. Still unchecked:
|
||||
fullscreen controls, hosting (firewall prompt), in-app update from one
|
||||
release to the next. The rest of the desktop tests are not in CI.
|
||||
|
||||
2. **Jellyfin** (epic, below).
|
||||
2. **Relay follow-ups** (see ARCHITECTURE "Relay"). First real use: owner's
|
||||
PC -> VPS -> friend, Jellyfin and a file. Then:
|
||||
- Source takeover: when the source leaves, another member with an
|
||||
account on that Jellyfin (or the same file: name, size, hash) takes
|
||||
over instead of the buffer running dry.
|
||||
- "Bob can't open this link — relay it?": offer the relay to whoever put
|
||||
a direct link up when a viewer's player fails to open it.
|
||||
- Show who is relaying and how fast in the member list; warn the source
|
||||
when its upload is below the media's bitrate.
|
||||
|
||||
3. **Smaller items**
|
||||
3. **Jellyfin** (epic, below).
|
||||
|
||||
4. **Smaller items**
|
||||
- Video output robustness: if media_kit's hardware path regresses to
|
||||
software (see ARCHITECTURE, Media: render mode), vendor a
|
||||
media_kit_video fix that builds the EGL display from GDK's native
|
||||
|
||||
Reference in new issue
Block a user