Durable Inference

An inference pipeline's steps depend on each other: authenticate, validate, call the LLM, stream tokens, record state. A queue can deliver each message, but it can't replay a half-completed workflow from a checkpoint when a process crashes mid-inference.

Temporal gives you durable execution: if the worker dies after the LLM returns tokens but before they're written to PostgreSQL, Temporal replays the workflow from the last successful activity. No manual retry logic, no dead letter queues, no state reconciliation code. The workflow definition is the retry logic.

The tradeoff is operational complexity - running Temporal is heavier than running RabbitMQ. But for a platform where durability is the core value proposition, that tradeoff is worth it from day one.

SignalR is excellent for .NET consumers - automatic reconnection, hub method routing, and built-in Redis backplane support. But it's a .NET-native protocol. If you're building a Python agent, a Rust CLI tool, or a Go service that consumes inference tokens, you don't want to pull in a SignalR client library just to read a stream.

Raw WebSockets are the universal transport. Any language with a socket library can consume the stream. The two transports share a unified event model - the same RelayEvent payload regardless of delivery mechanism. Consumers pick the transport that fits their ecosystem without sacrificing any functionality.

Redis Streams give you an append-only log with consumer groups - which is exactly what token streaming needs. Each tenant's inference output writes to a dedicated stream. Multiple consumers (relay instances) can read from the same stream without interfering with each other.

The alternative was piping tokens directly from the worker to the relay over gRPC or HTTP. That works until the relay instance isn't available - then you need a buffer, retry logic, and delivery guarantees. Redis Streams already provide all of that. The stream is the buffer, consumer groups handle delivery, and the data persists until explicitly trimmed.

Each message carries an XxHash3 integrity hash - lightweight, non-cryptographic, and cheap enough to verify on every read. If a message is corrupted or replayed incorrectly, the consumer detects it immediately.

An inference pipeline has ordered steps: pending, executing, streaming, completed. In a concurrent system with retries, two workers can both try to advance the same step. Without protection, you get duplicate token streams, double writes, or a step that skips ahead.

Compare-and-swap (CAS) makes each transition atomic - a step can only advance if its current state matches what you expect. If another worker already moved it, your CAS fails and you know to back off. Combined with cache invalidation on every successful transition, the system maintains coherence without distributed locks.

PostgreSQL is the most boring correct choice - and that's exactly why. Session state, inference history, step transitions, tenant configuration - all of this is relational data that benefits from ACID transactions, foreign keys, and a query planner with decades of production behind it.

The alternative was splitting state across multiple stores early - Redis for sessions, a document store for inference history, something else for configuration. That creates consistency problems between stores on day one. PostgreSQL handles all of it in a single transactional boundary. When scale demands splitting, you split with data to guide the decision.

The conventional wisdom is that infrastructure adds latency. Orchestration, durability, observability - every layer is supposed to cost you, and a workflow engine is supposed to be too complicated and too slow for a team that has to move fast.

Measured against the same model call made directly, over 200 runs per arm on two runtimes, Temporal adds 16 to 21 ms at p50, and its own time outside the call at p99 is 25 to 32 ms. The whole platform, with tenant isolation, distributed tracing and replay behind it, adds 35 to 46 ms at p50 on LM Studio, and the model call inside the worker takes as long as the direct one. The milliseconds are fixed, so a short answer carries a larger share: 5.9% of an 88-token answer for durability, and 11.5% of an 80-token one for the whole platform.