Every model call runs as a workflow, and a dead worker costs one retry.

Watch the top half first. The client calls an API, the API calls the model provider, and the connection stays open while the answer builds up. Four fifths of the way through, the model fails. The error travels back to the client, everything generated so far is thrown away, and the retry starts again from nothing.

The bottom half is Continuum, and the same failure hits it. The client handed the request to the Gateway and moved on. The prompt waits in a Redis stream, Temporal gives the work to a worker, and the answer builds up in a second stream. When the model fails, the worker clears what it had written, Temporal runs the call again, and the answer arrives whole.

Everything the request needs lives somewhere that outlives the process running it: the prompt and the output in Redis, the messages in Postgres, and the run's history in Temporal. A worker is only ever holding a copy.

The Console sends POST /v1/sessions/{sessionId}/infer with the prompt as the request body, up to 128 KB. The Gateway checks that the session belongs to the caller and is still active, and reads the body once, 8 KB at a time, into a pooled buffer. One read feeds two writes, issued together. The prompt goes into a Redis stream under {userId}:input:{messageId}, and into Postgres as the user's message, beside an empty assistant reply marked as streaming.

Then it starts the workflow inference-{messageId} on the inference task queue. The workflow carries ids and nothing else: the prompt never enters Temporal's history. The Gateway answers, about 12 ms after the request arrived, and the client is free before the model has seen a token.

Input and output streams share one format. The prompt goes in as entries of up to 8 KB, written from a buffer rented out of the shared pool and handed back, never a fresh array per chunk. The answer goes in as the model produces it: the first token in an entry of its own, then one entry for every 50 ms of tokens. A closing entry marks the end and records the total, and only then does the stream get its 24-hour expiry.

A swarm worker takes the activity from the task queue. Before it writes anything, it deletes whatever output an earlier attempt left under {userId}:output:{messageId}, and reads the prompt back out of the input stream. Then it makes one streaming call to the model. The loop reading the model only queues each token, and one writer drains the queue into the output stream: the first token at once, then whatever has arrived every 50 ms, with the model's reasoning in a stream of its own, {userId}:collating:{messageId}.

If the worker dies, Temporal still holds the workflow's history, and hands the activity to whichever worker is free. A crashed worker is noticed when its five-minute attempt runs out; an attempt that throws is retried five seconds later. The backoff doubles up to two minutes, and the retries keep coming for thirty minutes with no cap on the count. A provider error that can never succeed - a 4xx other than 408, 409 or 429 - stops them at once.

The next attempt uses the same key, so the answer's address never changes: anything that knows the message id knows where to read. The prompt stays in Redis until an attempt succeeds, so every retry reads exactly what the first one read.

A duplicate can't run twice either. Every submission mints its own message id, the workflow's id is that message id, and starting a workflow that already exists does nothing, so a start that is retried lands on the run already going.

When the answer is written, the worker finalizes the stream and posts a RelayEvent to the Relay: ServerInferenceCompleted, with the output's URL. The Relay pushes it to the session over a raw WebSocket and to the session's SignalR group at the same time, and the workflow marks the message complete in Postgres.

The client fetches GET /v1/inference/{messageId}/output. The first read comes from the Redis stream and clears it; after that, the completed message in Postgres answers. The Relay never carries the answer itself. It only says where to look, so a notice that goes astray loses nothing: the output is still where it was.

Chat is the second way back, built for the Dashboard, where a person reads the reply as it is written. The turn is handed off the same way: the Dashboard posts it to Chat, which saves the message beside an empty reply and signals the thread's long-running workflow, chat-{threadId}, and a swarm worker runs the model. The worker publishes each token to the thread's Redis channel, chat:{threadId}:channel, and the Chat host holding the Dashboard's connection subscribes to it and streams the tokens over its own SignalR hub, /v1/chat/stream. When the last one arrives, the Dashboard reads the finished message back from Postgres.

Chat doesn't use the Relay, because a reply is hundreds of fragments a second for one thread, and the Relay's path costs an HTTP POST and a group broadcast for every event. That suits the few notices a run sends. A streaming hub method answers only the connection that called it, and the host serving that connection subscribes to Redis itself, so Chat scales out with no SignalR backplane. A client that doesn't speak SignalR takes the first way back: the Relay's notice, then the Gateway's output.

The startup consensus is that a workflow engine is too complicated and too slow for a team that has to move fast. Measured apples to apples against the model call it wraps, durability costs 19 ms of a 1.9-second answer on LM Studio, and 18 ms of 3.1 seconds on vLLM, over 200 runs each.

• Client → model. The call made directly, timed from the request to the last token.
• Client → Temporal → model. The same call inside a workflow of one activity, with the answer returned as the workflow's result. Nothing of Continuum's is in the path: no Gateway, Postgres, Redis, Relay or fetch.

The bare test measures durability. The real-world run measures a request through the whole platform, which does a good deal more than durable inference. Between the submit and the answer in the client's hands:

• the Gateway checks the caller's auth and session, from the Redis cache;
• it writes the thread and both messages to Postgres, and the prompt to a Redis stream alongside them;
• it starts the workflow and answers 202;
• Temporal hands the activity to a worker, which checks the prompt is there, clears whatever a failed attempt left, and reads the prompt;
• the worker streams the answer into Redis as the model writes it, so a client can read it as it arrives;
• the workflow posts the Relay's notice as a local activity, which Temporal retries if it fails, and the Relay pushes it to the client over SignalR;
• the client fetches the output through the Gateway.

The obvious design hands the prompt to Temporal and lets the answer come back as the activity's result. It works: the bare benchmark above does exactly that, and a 700-token answer is about 3 KB, where Temporal warns at 256 KB per payload. But an activity's result only exists once the activity finishes, so Temporal can't be the pipe for tokens a client should read as they arrive.

It would also grow the history. Temporal writes every input and result into the workflow's event history, keeps it, and replays it whenever a worker picks the run back up, so a 128 KB prompt would be written into it on every run. Continuum learned that lesson in Milestone 1: carrying a chat's history in workflow state bloated the event log, and the fix was the same one. The workflow holds ids, and loads what it needs from the stores each turn.

Postgres is where the answer ends up, as the completed message, and it is the durable truth. It is the wrong place for the stream itself. A token arriving every few milliseconds would be a transactional write each time, with a new row version and a WAL record behind it, and a reader would have to poll to find out whether anything new had arrived.

A Redis stream is an append-only log ordered by entry id: cheap to add to, readable from any point, and it expires on its own a day after it's complete. Milestone 0 set the split at the start: Redis is the hot landing layer for streaming input and output, and Postgres takes one write when the answer is done. Continuum had already moved a single, frequently written timestamp out of the sessions table for the same reason.

Redis was in the stack from the first day, as the cache, and it went on to carry the SignalR backplane and the Pub/Sub channel that live tokens travel on. A stream per message is one more key on a server the platform already runs, with the expiry and the ordering built in, and nothing new to deploy.

It was measured before anything depended on it. The first BenchmarkDotNet project, in the first week, timed the stream writer and reader, and set the chunk size. The first version used 64 KB, as the plan said, and the next 2 KB. The benchmarks showed write time tracking the number of stream appends rather than the bytes, so the writer settled on 8 KB.

• Borrowed buffers. The Gateway reads a prompt into a pooled buffer, and the stream writer rents its 8 KB buffer from the shared pool and hands it back, so a busy Gateway doesn't turn every request into garbage for the collector. The field names every stream entry carries are allocated once, when the process starts, never per entry.
• Bounded reads. A reader asks Redis for eight entries at a time, so a single read never holds more than 64 KB, however long the stream.
• Tokens, coalesced. The token path is the hottest in the system. A pass in April cut its allocations, and until September the worker still awaited a stream entry and a Pub/Sub message for every token. In the inference activity, one writer now drains them every 50 ms, sending each chunk's stream entry and message together. At the model's pace a 1,000-token reply is 50 entries instead of 1,000 and allocates 206 KB instead of 2.1 MB, and fetching a 730-token answer fell from 46 ms to 5. In a burst, 1,000 tokens are written in a median 2.8 ms, against 1,025 ms with a write and a publish per token.
• Redis memory gives itself back. The input stream is deleted when an attempt succeeds, the output stream is purged after the client's first read, and anything left expires 24 hours after it closed. Postgres keeps the answer.
• Temporal's history stays small. A workflow carries ids, never payloads, and a chat's long-running workflow continues as new every 100 turns, carrying any prompts still queued, so no history grows without end.
• More workers, more capacity. Workers poll their task queues, so another worker is more capacity with no configuration; the only setting a fleet has to change is the dev server's single task-queue partition. Relay instances scale out behind SignalR's Redis backplane, and each one caps itself: 10,000 connections, 10 per session, a keepalive every 45 seconds, and a load factor it reports. Chat hosts need no backplane, since each subscribes to the Redis channels of the threads its own connections are streaming.
• Nothing on the request path waits for a model. Submitting is three Postgres statements, the prompt into Redis alongside them, and a workflow start, with the body capped at 128 KB: 12 ms to the 202, whatever the model is doing.

Milestone 0 built it in a month, in dependency order, starting with the part everything else would sit on: the Redis stream reader and writer, measured with BenchmarkDotNet before anything used them.