# Architecture Plain overview first: [HOW-IT-WORKS.md](HOW-IT-WORKS.md). Wire format: [PROTOCOL.md](PROTOCOL.md). Plans: [ROADMAP.md](ROADMAP.md). ## Packages | Path | What | Rule | |---|---|---| | `packages/core` | protocol messages, sync engine (`SyncEngine`), constants, config, Range parsing, logging, Jellyfin client | pure Dart, no Flutter | | `packages/server` | Relic HTTP + WebSocket server (`SyncServer`, `ServerHub`, `Session`, media endpoint, relay buffer, Jellyfin relay), the `serve()` entrypoint, `bin/server.dart` (dedicated server) | pure Dart, no Flutter | | `apps/desktop` | Flutter app: UI, `SyncClient`, `SessionCoordinator`, `PlaybackController` (media_kit/mpv), `ServerIsolate`, updater | ships with `watch-together-server` | | `tool` | harnesses: `range_logger`, `shaper`, `fake_viewer`, `e2e_check` | dev only | | `deploy` | systemd unit for the dedicated server | | Guard: `packages/core/test/flutter_free_test.dart` fails if anything under `core/lib` imports Flutter, `dart:ui` or `signals_flutter`; CI runs analyze + tests for core and server on every push. Server has no such test but the same rule; CI checks it with `dart compile exe` of `bin/server.dart`. The client WebSocket (`web_socket_channel`) lives in `apps/desktop`; Relic has no client. ## Server lifecycle One entrypoint: `serve(config, events:)` in `packages/server`. `config` is an `AppConfig.toMap()` (primitives only, so it crosses isolates). It starts a `SyncServer` (bound to all IPv4 interfaces) and reports on the `events` port: `ready` (with the port and a `stop` SendPort), `error` (start failed), and every log record as `log`. Sending `{'t':'stop'}` to the `stop` port closes all sockets and returns, and the isolate exits. `{'t':'file', 'path', 'reply'}` on the same port makes a local file the hosted one and answers its name and size (`ServerIsolate.setFile`): the host's in-session File swap then sends a `swap` with that name and size, which the server matches to the file. - **Hosting from the app**: `SessionState.host()` spawns an isolate running `serve` (`ServerIsolate`), waits for `ready`, then joins it over `127.0.0.1` like any other client. Server logs go to the app's log with a `[server]` prefix. The isolate runs with `errorsAreFatal: false`: an uncaught async error is logged and the server keeps going. If the isolate dies anyway, the session shows "server crashed: ..." and leaves; the shell says so in a dialog (during setup the host attempt fails with it). Stop (leave, close, cancel during setup, a failed own join) sends `stop`, waits up to 3 s for exit, then kills the isolate. - **Dedicated server**: `bin/server.dart` compiled with `dart compile exe` and shipped next to the app as `watch-together-server`. It parses flags (`--port`, `--secret`, `--name`, `--media`, `--advertise-host`, `--jellyfin `, `--[no-]yt-dlp`, `--yt-dlp-dir`, `--config`; default port 8080) into the same map and calls `serve`; SIGINT and SIGTERM stop it. A Flutter binary cannot run without a display, which is why the server is its own build. `SyncServer` runs a 1 s heartbeat: stale sweep, closed-socket sweep, silent socket drop, state + feed broadcast. There is one session, id `default`. Passwords: the default secret is empty, which means an open server (a warning is logged). Otherwise one shared secret, compared at hello. Host: when hosting, the app generates a random `hostKey` per hosting, puts it in the server config and sends it in its own `hello`. The member whose key matches is the host (`members[].host`: the crown, art rights). The address a connection comes from plays no part (so a reverse proxy or tunnel on the same machine does not make everyone the host). The dedicated server has no key, so it has no host. Nobody removes anyone: members sit each other out ("Sitting out" below), the host included, since the hosted server and its `/m/` file serving do not need the host's player. ## Sync model Loosely modeled on Syncplay, with a different readiness design. - **Transport** is `playing`, `held` or `finished`. The server owns the clock: `positionAt(now) = position + (playing ? now - serverTime : 0)`. Reports from clients never move it. Only seek, swap, gate transitions and the end of media write `position`/`serverTime` and bump `eventId`. - **Clock offset**: `serverTime` is on the server's clock. Each client sends a `ping` with its clock every second and estimates the offset NTP style (`ClockSync`: `offset = s - (t0 + t3) / 2` of the sample with the smallest round trip among the last 16). All extrapolation uses `SyncClient.serverNow()` (local clock plus offset), so a client whose clock is off still lands on the server position. - **Reports**: clients send `position`, `buffered`, `eventId` every 1 s, plus the media `duration` and its `mediaId` once the player knows it. The server stores `buffered` and liveness, and takes the first reported duration for the current media (swap resets it); a report with `eventId` older than current has its position ignored. - **Ready gate** (HELD to PLAYING): the in-set is non-empty and every in-set member is ready, fresh and has `buffered >= need`. `need` is 4 s (`minBufferSecs`), or near the end what is left minus 2 s (`endSlackSecs`, floor 0), since mpv cannot buffer past the end. So a lone viewer plays alone; with others present everyone must ready. - **In-set** (who the gate waits for): a member joins it on joining, or on coming back from stale, while not PLAYING (any media, the one the server started with included); joining while PLAYING it stays out (`ready=false`) until it readies: a late joiner never pauses anyone. A swap puts every fresh member in the set and keeps everyone's ready flag, so the next video starts once everyone ready has buffered (the client setting "Unready when the video changes", `AppSettings.unreadyOnSwap`, default off, sends `ready=false` itself on a new `mediaId`; not for the first state of a session, and a reconnect does not restore a ready that a swap meanwhile would have dropped). Stale, leave and sitting out remove from the set. - **Sitting out** (`EngineMember.out`, `state.members[].out`): a member stays connected (chat, feed, party, nudges) but is never in the set and never ready: no `ready` or `force` reaches it, a swap leaves it out, its reports keep it fresh but their `buffered` counts as 0. Anyone can sit anyone out (`out` with `out:true`, the host included); only the member itself comes back (`out:false`): it is then in the set like a fresh join (unless PLAYING) and still unready. The app: an out member's player has nothing open (no stream or relay pulls; a relay it serves goes on) and shows the join-back card (cover, title, episode, the server position and playing/paused). "Join back" or Space sends `out:false` and then `ready`: at once while HELD (the gate waits for its buffer), while PLAYING once the player has buffered `need` (at most 15 s), so the others do not pause for the load. Putting something up while out comes back first. A reconnect that lands as a new member sits out again. - **Pause**: while PLAYING, any in-set member who becomes not ready, under `need` buffered or stale flips the server to HELD (feed action `pause` by `server`). Recovery resumes by itself when the gate passes. - **End**: while PLAYING with a known duration, reaching it sets FINISHED (feed `finished` by `server`, position = duration). Clients pause where they are, except one within 3 s of the end (`finishSlackSecs`), which plays out to mpv's own stop; at the end the position shows the full duration. FINISHED never auto-resumes. A seek turns it into HELD at the new position (resuming at once if the gate passes); a swap resets to HELD. - **Seek**: allowed from any member in any transport; the server writes the position, bumps `eventId` and records who sent it (`seekBy`, `seekSeq` in `state` for that event); flags are untouched. Clients diff each `state` broadcast by `eventId` (`StateDiff`): new event means seek to the computed server position, transport change means play/pause. The coordinator applies states one at a time, newest first (a state arriving meanwhile replaces any still waiting): interleaved applies could land an older pause after a newer play. Between events: drift over 5 s for more than 2 s seeks, ahead by more than 1.5 s sets rate 0.95 until level, otherwise nothing. - **User seeks**: the player is a `SyncPlayer` whose public `seek` is the user's: the session's seek bar (`ChapterSeekBar`, one `seek` per pointer move while dragging and one on release, like media_kit's), the arrow keys, skips, chapter jumps and `seekTo` all become explicit events. The player moves at once (scrub preview); `UserSeekGate` sends one `seek` intent with a fresh `seq` after 300 ms without another request (`seekDebounceMs`). While the user is seeking, and until the echo of the last sent seq arrives (3 s at most), server seeks never touch the player; the echo of our own seek is not applied either (the player is already there). A server seek that has not landed after 1.5 s is re-applied once (`_checkSeekLanded`), to the server position recomputed now, and only while its event is still current and the user is not seeking. - **Liveness**: WS ping every 5 s; silent over 15 s means stale (out of the gate, `stale` feed); over 30 s the socket is closed and the member removed. A stale member goes back into the set with its next frame if it is ready (also one stale at a swap) or the server is not PLAYING. Closed sockets are polled with `isClosed` every second: Relic's server-side `onDone` does not fire for dead peers, so presence never relies on it. - **Resume**: a reconnecting client sends its previous token as `hello.resume`. If the server still has that member (an unclean drop it has not noticed), the new socket takes it over: same token, member id, ready flag and media URL (the player keeps streaming), no join/leave feed, and the dead socket is closed without removing the member. An unknown token joins as a new member. - **Client policies**: no state broadcast for 5 s means connection lost, clear the server state and reconnect every 1.5 s until cancelled (an epoch counter makes a cancelled attempt stale). After a reconnect the ready flag is restored from the server's state. Cancel during setup aborts the join. Closing the window leaves gracefully (2 s timeout). - **Idle** (`IdleTracker`): measured from the later of the last input and the moment playback stopped. While HELD and self ready: warning at 8 min, auto-unready at 10 min, and never less than 2 min after the warning. With "When idle, sit out instead of just unready" (`AppSettings.idleSitsOut`, default off) the 10 min step sits out. - **Nudge**: `nudge` names a member; the server passes it to that member only (no feed), at most 3 per second and 20 per minute per sender (`nudgesPerSec`, `nudgesPerMin`; the rest is dropped silently). The nudged app plays the nudge sound whatever its per-sound switch (the volume still applies), toasts "Alice nudged you" and asks the desktop for attention when its window is not focused: the runners' `starsync/window` channel (`attention`), the urgency hint on Linux (GTK, cleared on focus) and `FlashWindowEx` until foreground on Windows. - **Deviations from Syncplay**: clock never follows the slowest watcher; no pause-on-leave; anyone can force ready/unready (all or one); presence gate; the clock is server time (clients correct for their offset to it). Additive wire changes only; bump `proto` and PROTOCOL.md in the same commit. ## Media Three sources, same sync. Any of them can also go through the relay (below). - **Link**: the media field holds an http(s) URL. It travels as `media.url` in `swap` and `state`, and as `mediaUrl` in `welcome`. Each app plays it directly (a web page through mpv's yt-dlp hook, see "yt-dlp"); the server is only the clock. A pasted link that `parseJellyfinLink` recognises goes up as a Jellyfin ref instead when the ServerId is known (in the link, else `/System/Info/Public`), so its `api_key` is never shared (`mediaForLink`, host and swap alike). - **Host local file**: `SyncServer.setLocalFile` / `mediaPath` makes the file the session media. `GET /m///` serves it with a Range table, gated by the viewer's token. The server builds the URL per viewer (`http://:/m/default//`), so tokens never reach other viewers. `` is the authority the client dialled; `advertiseHost` (default empty) only overrides it when set explicitly. When a `state` broadcast has no `media.url`, the client builds the URL itself from the host it connected to. Bytes stream file to socket and are counted as streamed (`bytesServed`); nothing is stored. The host plays through loopback like everyone else. - **Jellyfin item**: the media is a ref (`{serverId, itemId, ...}`, wire in [PROTOCOL.md](PROTOCOL.md) "Jellyfin media"); the server never stores or sends a URL or token for it. Each app resolves it with its own account and direct-plays the static stream; guests use the server's own login when it has one ("Jellyfin relay"), else the member who put it up ("Relay"). Range table (`resolveMediaRequest`, tested byte-exact): | Request | Response | |---|---| | no Range | 200, `Content-Length` = size | | `bytes=a-b` | 206, `Content-Range: bytes a-b/size` | | `bytes=a-` | 206 to EOF | | `bytes=-n` | 206 last n bytes | | unsatisfiable | 416, `Content-Range: bytes */size` | | host has no file | 503 + `Retry-After: 2` | | bad / unknown token, unknown session | 401, 410 | Always `Accept-Ranges: bytes`; `Content-Type` from the extension (`contentTypeForFile`). Relic hands the Range over as a typed `RangeHeader`; the endpoint takes the first range and keeps the unit, so non-`bytes` units still get 416. What mpv does against it (recorded with `tool/range_logger.dart`): mp4 opens once with no Range then streams `bytes=0-` to the end; mkv opens `bytes=0-` twice (probe, playback), reads a small tail at `size-n` (cues, expressed open-ended), then every seek is a fresh `bytes=N-` connection. A server outage of a few seconds is ridden out from mpv's cache with no error. Longer outages exhaust the cache and rely on reconnect (not chaos-tested yet). mpv settings (`PlaybackController`): `Player` with `bufferSize` 256 MiB (media_kit's `bufferSize` sets both `demuxer-max-bytes` and `demuxer-max-back-bytes`), `libass` on; pinned via runtime property: `demuxer-readahead-secs=150`, `cache-secs=150` (about two minutes of read-ahead: rides out short drops, intro skips are instant), `demuxer-max-bytes=268435456` (150 s up to ~14 Mbit/s; a 1.4 GB / 24 min file is ~8), `demuxer-max-back-bytes=67108864`, `stream-lavf-o=reconnect_streamed=1,reconnect_on_http_error=1,reconnect_delay_max=4` (comma form; this mpv build rejects colon-joined). The cache is in RAM: media_kit sets `cache-on-disk` without a `cache-dir`, and mpv falls back. mpv ignores invalid values silently, so every pin is read back and a mismatch is logged and kept in `pinMismatches`. Measured against the `/m` endpoint (`readahead_e2e_test`): a paused player buffers ~147 s ahead (10 s before). media_kit has no public `setProperty`/`getProperty`; the code calls them through a guarded dynamic call on `player.platform`. Hardware decoding comes from Settings (auto, off, custom) and is passed to the `VideoController`; env `SYNC_RELAY_HWDEC` (e.g. `no`) overrides it. It is read per player, so a change applies on the next join or Reload video. Shaders (per viewer, local only, no wire change): mpv `glsl-shaders`, set through the same dynamic `setProperty` and read back (`PlaybackController.applyShaders`). The value is the platform path list (`:` joined, `;` on Windows). One saved choice in `AppSettings` (`shaderPreset`, `shaderQuality`, `customShaders`): applied on every player creation (join, Reload video) and live when changed from the session (the control bar's shader menu, Ctrl+0..6, both also save it). The Anime4K v4.0.1 files live in `apps/desktop/assets/shaders/anime4k/` (only the 14 the presets use, plus its MIT LICENSE); mpv needs real paths, so `extractAnime4k` copies them once per version to `/shaders/anime4k-/` (a `.complete` marker is written last). `anime4kFiles` maps each mode to the file list of Anime4K's mpv templates: HQ = the high-end GPU set (VL + M networks), Fast = the low-end set (M + S). A missing file, a path containing the list separator, a read-back mismatch, or an mpv `libmpv_render` error while shaders are on turns shaders off (saved) with a toast. Render mode: media_kit's Linux/Windows video output creates mpv's render context (OpenGL, else its software renderer) when the `VideoController` is created, and nothing in its API says which. The player therefore requests mpv logs at `v`: OpenGL init logs `GL_VERSION` under `libmpv_render` (`RenderMode.gpu`; "suspected software renderer" means llvmpipe, `softwareGl`); none within 1.5 s of the controller being ready means the software path (`software`, where shaders do nothing; the shader toast, the menu and Settings say so). After startup the `v` level adds no steady traffic (measured: 0 messages over 5 s of playback). The hardware path is fragile: media_kit borrows whatever EGL context GDK left current on the main thread (upstream media-kit#1404), so an X11 session or a Flutter/GDK change can silently fall back to software. Player lifecycle: the `Player` and its `VideoController` live for one session: created on join, disposed on leave, replaced by `recreate()` for "Reload video" (button or Ctrl+R; volume kept). `openUrl` opens with `play: false`; play/pause/seek/rate come only from `StateDiff` actions, the play gate (`localPlayAllowed`: self in-set and ready) or the user's own seeks (see Sync model). A server seek within 0.5 s of the current position is skipped. `leave()` stops the player (otherwise audio keeps playing). On media change (`mediaId` bump) the coordinator opens the new URL and `StateDiff` seeks to 0, holds and resets rate. Frozen picture: `FrameWatchdog` samples about once a second. It hashes a small capture of the compositor's centre (`ShownFrameProbe`) and suspects a freeze when position advances while that hash stays constant for 3 s. Two differing mpv `screenshot-raw` frames show the decoder is still producing pictures, which confirms the freeze (a still scene is dismissed). A confirmed freeze reloads the video with a toast. It fires once until the picture moves again for 10 s, so it cannot loop. Seeks, pause, buffering and a hidden window disarm it. Chapters, segments, tracks (`PlaybackController` signals, values from `playback/media_marks.dart`, no widgets; any controls UI reads them): - `chapters`: mpv `chapter-list` (observed through the same dynamic `observeProperty` and read as JSON): `MediaChapter` index, title, start; end = next start, the last one's = `duration`. - `segments`: the Jellyfin item's MediaSegments (`GET /MediaSegments/{id}`, 10.10+: Intro, Outro, Recap, Preview, Commercial), fetched by the coordinator with `OwnPlay` only; relay and blocked viewers cannot call Jellyfin and use chapters alone. Empty without a segment provider plugin (the case on the owner's 10.11 server). - `tracks`: audio and subtitle `MpvTrack`s from mpv `track-list` (id, language, title, codec, channels, default / forced / hearing-impaired / external, `selected`), re-read after every selection. - `position` (250 ms steps), `duration`, `buffered` (end of the cached range: mpv `demuxer-cache-time`, media_kit's `buffer`, 250 ms steps); computed `opening`, `ending` (`SkipRange`) and `currentChapter`. Skip rules (pure, `media_marks.dart`): - Opening = first Intro segment, else the first chapter whose title has a whole word OP / Opening / Intro (any case, digits allowed: "OP1"). Ending = first Outro segment, else ED / Ending / Credits / Outro. - `skipOfferAt` (the coordinator's `skipOffer`, null at `finished`): inside the opening (up to its last second) "Skip OP" to its end; inside the ending "Skip ED" to its end only (no guessing about previews: the viewer decides). Without any opening, without chapters (a chaptered file without an OP chapter has none on purpose) and with media over 20 min, "Skip 1:30" during the first 6 min. - +1:30 (`skipForward`: the pill, the controls' "1:30" button, Shift+Right) is a `nudge`, so repeated presses add up. - Chapter steps (PgUp / PgDn, `chapterStepTarget`): forward to the next start; back to the current start, or the previous one within 5 s of it (mpv's `chapter-seek-threshold`). - Every skip and chapter jump is a user seek (debounced, everyone follows). Never unmount GPU-textured widgets (avatar images) on a transport-gated branch: teardown races mpv's frame upload and the window goes black. Keep them mounted and skip painting (`Visibility` with `maintainState`). ## Jellyfin client `packages/core/lib/src/jellyfin/`, pure Dart over `dart:io` `HttpClient` behind a `JellyfinHttp` interface (tests inject a fake). Plans: [ROADMAP.md](ROADMAP.md) "Epic: Jellyfin". - `JellyfinClient`: one server + one user. Auth (Quick Connect, password, `/Users/Me` check, logout), browsing (views, Resume, NextUp, Latest, paged `/Items` with `DateCreated`, search, seasons, episodes, recently played), image URLs (tag + maxWidth, no token needed), PlaybackInfo, stream URLs, MediaSegments (a 404 = none), Playing / Progress / Stopped reports, played / unplayed / resume position. - `normalizeServerUrl`, `parseJellyfinLink`: pasted addresses and item links -> `(baseUrl, itemId)`; accounts are keyed by `ServerId` (`/System/Info/Public`). - `episodeAfter`: next episode is the successor in `/Shows/{id}/Episodes` (crosses seasons, skips other copies of the same episode); NextUp means first unwatched, not next-after-X. - `matchTrack` / `matchTracks`: carry audio/subtitle choice to the next episode (language required, then title words, forced/HI flags, codec, channels). Gotchas (Jellyfin 10.11): - Every request sends `Authorization: MediaBrowser Client, Device, DeviceId, Version[, Token]` (values percent-encoded). Unauthenticated calls missing a field get 400 with no body. - A POST must carry `Content-Length` even when empty: behind nginx a body-less POST without it gets 400 (this was the Quick Connect Initiate failure). - Jellyfin overwrites a token's stored device name and app version with the header's. Only use tokens this app obtained itself; a relay using a viewer's token must send that viewer's client fields (the server relay sends exactly its credential's fields). - Static streams (`/Videos/{id}/stream?static=true`) serve the original file with byte ranges. The share URL has no credentials; each side adds its own `api_key` (`authorizeUrl`). - Ticks: 10,000,000 per second. ## Jellyfin relay For viewers without an account on the item's Jellyfin (`packages/server/lib/src/jellyfin_relay.dart`). The server may hold one `JellyfinCredential` (config key `jellyfin`: `baseUrl`, `serverId`, `token`, `client`, `device`, `deviceId`, `version`; primitives, so it crosses the isolate boundary in the `serve()` map). The hosting app fills it with its own app-obtained token and that token's device fields; the dedicated server reads it from `--jellyfin ` (YAML/JSON with those keys, keeps the token off the command line) or a `jellyfin:` map in `--config`. An empty `serverId` is looked up at start (`/System/Info/Public`); if that fails the relay stays off. The token is never logged (`toString` and config parse errors leave it out). - Available (`state.relay`) only while the current media is a ref whose `serverId` equals the credential's. Only that item is relayed, only to a valid viewer token: no open proxy. - Each request opens `/Videos/{id}/stream?static=true&mediaSourceId=` with `Authorization: MediaBrowser` built from exactly the configured fields (Jellyfin rewrites a token's device fields from the header), forwards `Range` untouched and `Accept-Encoding: identity`, and returns Jellyfin's status and range headers. Bytes stream through (no buffering) and count toward `bytesServed` as sent. A viewer disconnect cancels the upstream response, which closes that connection (tested: `HttpClientRequest.abort` does nothing once the response has arrived; cancelling the response subscription does). - No progress reporting in the server: watched state is per client. ## Relay A member's media through the server's memory, for viewers who cannot reach it themselves (`packages/core/lib/src/relay.dart` for the wire and the source, `packages/server/lib/src/relay_buffer.dart`, `relay_routes.dart`; wire in [PROTOCOL.md](PROTOCOL.md) "Relay"). What gets relayed (`SessionCoordinator.swapPicked`): | Put up | Others get it | |---|---| | File, by the host | `/m` from the hosting app (no relay) | | File, by anyone else | relayed from that member's computer | | Link, "Everyone fetches it" (default) | each app fetches it | | Link, "Server fetches it" | the server reads it once into the buffer; the link is never broadcast; refused on an open server (it would fetch any address it can reach for anyone) | | Link, "Through me" | relayed from that member's computer; for links only they can open | | Jellyfin item (browser pick or pasted link) | own account first, then the server's login (`/j`), then relayed from the member who put it up (with their sign-in, or the pasted link's key, which stays on their machine) | Relayed links are range-read: a media file as is, a web page as yt-dlp's formats, resolved where the bytes are fetched (see "yt-dlp"); the swap carries only the page URL and the mode. A page's formats are one file, or a video-only and an audio-only file (YouTube has only those): a media of two **tracks**, each relayed as its own byte stream with its own buffer and URL, played together by viewers as an mpv EDL (`edl://!no_clip;%n%