# Live STT WebSocket protocol -- `wss://.../v1/stream` OpenAPI can't express WebSockets; this document is the contract for the stt-proxy Worker's live-transcription relay. Implementation: `workers/stt-proxy/src/index.js` + `translate.js`, `dial-phase-buffer.js`, `planned-close.js` and `upstream/byteplus.js` (keep in sync). Consumers: the desktop's `WorkerProxyAdapter` (`src/main/agent/stt/`), future React Native capture. ## Connect ``` wss://seameet-stt-proxy-dev.seameet.workers.dev/v1/stream?lang= Authorization: Bearer ``` Auth is the standard `Authorization` header on the upgrade request -- Node/Electron and React Native WebSocket clients can set it; **browser `WebSocket` cannot set headers**, so the web library does NOT do live STT (by design -- capture is a device feature). Verification: ES256 against the project JWKS with HS256 fallback. Entitlement: same `checkEntitlement` as sync-api (plan `sync`, active/trialing, not ai_disabled); failures close the socket immediately after upgrade. The STT provider (Soniox `stt-rt-v4` / Deepgram `nova-3` / BytePlus Seed-ASR `bigmodel`) is chosen server-side from the `provider_routing` table -- clients never pick. The `lang` query hint participates in routing (`match_lang` rules). ## Client -> server frames | Frame | Encoding | Meaning | |---|---|---| | Binary | PCM 16-bit LE, 16 kHz, mono | Audio; send in ~100-250 ms chunks | | Text `{"type":"finalize"}` | JSON | Flush upstream + emit remaining finals; server closes when drained | ## Server -> client frames (JSON text) | `type` | Payload | Meaning | |---|---|---| | `ready` | `provider` | Upstream relay connected. Audio sent before it is not lost: the server holds the most recent 10 s and replays it in order, so a client may stream from open. Time 0 is the first frame the client sent (unless the dial outlasted those 10 s) | | `segment` | `segment: TranscriptSegment` | One transcript segment. `segment.isFinal` distinguishes interim (replaces the previous interim) from committed text. Shape contract: `src/main/agent/stt/STTProvider.js` -- `{text, speaker?, language?, confidence?, startMs?, endMs?, isFinal}`. `startMs`/`endMs` are ms from this connection's time 0. Every provider sets `startMs`; on BytePlus the utterance already in progress when the session opened (the first one after a seam) arrives with `startMs: 0` | | `error` | `error` (code string) | Fatal; socket closes next. Notable codes: `all_providers_unavailable` | | `relay_roll` | -- | Planned roll; close 1012 `relay-roll` follows. Reconnect at once -- not a failure | | `session_too_long` | `capMs` | Per-session cap reached; close 1011 follows. Terminal: don't reconnect | | `free_exhausted` | `capMs` | Free transcription pool spent; close 1000 follows. Terminal: don't reconnect | ## Lifecycle & limits - One session = one recording. Sessions are metered into `usage_events` (`feature: 'stt'`, `metadata.path: 'managed'`; Soniox sessions bill by the documented token conversion, Deepgram per second). - Reconnect policy has two layers, and they are deliberately different speeds: - **Transport (`WorkerProxyAdapter`)**: one silent reconnect, 1 s after an unexpected mid-session close, invisible to the renderer; it is available again once a connection has held 60 s. A `relay_roll` reconnects at once without spending it. - **Session supervisor (renderer, `src/utils/sttReconnectPolicy.ts`)**: if the session is still dead after that, it retries while the recording is running and not paused, on a **slow** ladder -- 5 s / 15 s / 30 s, then 60 s. Slow by design: each attempt consumes a concurrency slot a zombie predecessor may still hold for its 120 s KV TTL, so fast retries make the outage worse. A `too_many_concurrent_sessions` rejection backs off a full 120 s (past the slot TTL). Terminal server states (free pool spent, 5 h cap) stop retrying. The BYOK Gemini Live path, which has no server slot accounting, uses the fast ladder instead (2/4/8/16 s, 30 s cap). The transcript panel also shows a manual **Reconnect** button whenever the session is `disconnected` or `error` and no hard wall is up; it is single-flight with the supervisor. - Reconnect resumes as a new provider session -- no replay -- but the desktop keeps the existing transcript and re-anchors incoming timestamps to the recording's timeline (`sessionTimestampOffsetMs`), so captions stay continuous across a drop and across pause/resume rather than restarting at 00:00. The adapter shifts each new socket's times by the audio sent before it, so its own reconnects stay on that timeline. At every seam, web and desktop first commit the in-flight interim as a final; otherwise the next session's first interim replaces it and the words are never saved. - **Planned closes** (`planned-close.js`): the control frame, then the close. Rolls come on a per-environment interval (`STT_RELAY_ROLL_MS`, 30 min by default; 180 s before 2026-10-01) and whenever the in-session re-check (every 3 min, unless the roll is that frequent) sees the kill switch turn on, the plan change, or an operator restart (after a rollback, so open sessions move onto the live version); the reconnect then meets the gate's new verdict. On BytePlus's finals-only path (`lang=auto` and languages outside its zh/en set) the server first flushes the unfinished segment for up to 4 s: segments keep arriving, and audio sent meanwhile is dropped. A client that hasn't completed the close handshake 10 s after the close is dropped anyway. - A close 1000 with **no** control frame before it means the server ended the session unplanned, usually because the provider session closed or errored (e.g. BytePlus's 8 s no-audio timeout). Reconnect for a fresh session. - Audio is sent in ~250 ms batches (renderer resample worker, `BATCH_INTERVAL_MS`), not per AudioWorklet render quantum. - Concurrency is capped per provider account and metered per uid by the proxy (KV slot, 120 s TTL, heartbeat-refreshed); `GET /v1/health` exposes live session counts (see `openapi.yaml`). The `/v1/file` upload lane shares the bucket, so a stuck slot also blocks the post-stop summary upload.