Most of the reader's saving was the benchmark's.
The first week of Continuum was a Redis stream writer, a reader, and a BenchmarkDotNet project to hold them to account. The reader's allocations fell by up to 43%, and most of that was the benchmark.
Week one
The first thing Continuum built was a way to put bytes in Redis.
Before there was a Gateway, a workflow or a model, there was a Redis stream. Every prompt Continuum would ever run had to land somewhere a worker could read it back, so the first week went on two small classes: a writer that takes a request body and appends it to a stream in chunks, and a reader that pages the chunks back out.
The code was written with Claude - Opus, running inside GitHub Copilot in Visual Studio - and the brief arrived in three parts, at volume.
Tristan
STREAM IT.
Tristan
BUFFER IT.
Tristan
DO YOU EVEN BENCHMARK, BRO?!
Claude
My silent despair begins again…
There was a lot of that. The third one became a project of its own on the first day,
Continuum., with BenchmarkDotNet's memory
diagnoser on every run.
Borrowed time
The first writer rented a buffer, then copied out of it anyway.
The rule for the hot path was to borrow memory rather than allocate it: rent a buffer
from ArrayPool, fill it from the request, give it back. The first version
did rent one, 64 KB of it. Then it copied every chunk into a brand-new array before
sending it to Redis.
// 18 February: the pool is there, and every chunk gets a fresh array anyway
var chunkBuffer = new byte[read];
Buffer.BlockCopy(buffer, 0, chunkBuffer, 0, read);The fix hands Redis a slice of the rented buffer itself, and it is still how the writer works today.
// today: a view onto the rented buffer, no copy
var entries = Helpers.GetNameValueEntries(buffer.AsMemory(0, read), totalBytes);
await _database.StreamAddAsync(uniqueId, entries).ConfigureAwait(false);The chunk size took three tries: 64 KB in the first version, as the plan said, 2 KB an hour later, and 8 KB on 23 February. At 2 KB, an 8 KB payload took five appends to Redis, and write time tracked the number of appends rather than the bytes. The writer settled on 8 KB, which lands most prompts in a single entry plus the one that closes the stream.
One version of the reader asked Redis for four entries and stopped there. Its test passed, because the test wrote a single entry. The paging loop arrived three minutes later.
Measure twice
Most of the week's saving was in the benchmark.
By the end of the week the journal recorded a win: read allocations down 23 to 43% across payload sizes, credited to the reader's fixes, among them field names cached once instead of built for every entry. The numbers hold up, but a second look in September found the drop at each size was the payload size plus about 70 bytes.
| Payload | Allocated, 23 Feb | Allocated, 24 Feb | Saved |
|---|---|---|---|
| 1 KB | 4.59 KB | 3.52 KB | 1.07 KB |
| 2 KB | 6.59 KB | 4.52 KB | 2.07 KB |
| 4 KB | 10.59 KB | 6.52 KB | 4.07 KB |
| 8 KB | 18.59 KB | 10.52 KB | 8.07 KB |
A cached field name saves the same few bytes whatever the payload, so it can't
produce a saving that grows with it. Something the size of the payload was going
away. The benchmark had been creating a MemoryStream the size of the
payload inside the method it was measuring, and that line moved into the benchmark's
setup on the same day. It took its allocation with it.
The runs had a second catch. They ran with Visual Studio's profiler attached, five iterations each, and each iteration took a fraction of a millisecond, so the timings swing wider than their own means. The allocation column is the one to trust. The reader did get leaner that week, from 27.4 KB per read of an 8 KB payload on 18 February to 10.5 KB, but most of that fall came from the harness and the bigger chunk, not from the reader.
What stuck
The shouting became a rule.
The writer still rents 8 KB from the pool and hands Redis a slice of it, the field names are still allocated once, and the reader still pages eight entries at a time. Offsets and an XxHash3 hash on every chunk came in March, when the same streams started carrying real prompts.
In April the engine's rules gained the line the whole first week had been shouting: "Benchmark before assuming," ending with "Gut feel is not evidence - numbers are." The first week adds a clause to it: the benchmark is code too, and a number from it is only as good as the line that produced it.
How those streams carry every inference today is on the durable inference page.