We Had to Build It Ourselves

Disaggregated inference is a great idea. Separate the routing from the compute. Let the scheduler see the full cluster. Move KV context instead of recomputing it. The architecture is sound.

We thought it would be even greater if it was built simply.

NVIDIA’s Dynamo coordinates inference engines such as vLLM, SGLang and TensorRT-LLM. It separates cluster orchestration from the engine doing the GPU work.

We chose to build both parts ourselves.

We’ve spent two decades operating production infrastructure, compiling custom Linux kernels, building network security, chasing down failures in systems that looked fine on paper. That experience shaped what we wanted here: a small set of components to deploy and a serving path we could inspect and change ourselves.

A GPU engine, with a routing binary when the deployment needs a cluster.

So we started writing

We opened an empty Rust workspace.

The forward pass came first. Then attention. Then the KV cache. Then continuous batching. Then the token sampler and kernel compilation. We implemented those paths in Rust and GPU kernels, without a Python runtime in the serving path or another inference engine underneath.

It’s a strange feeling, writing your own attention kernels when perfectly good ones exist in open source. There were mornings where it felt like stubbornness. But there’s a clarity that comes from being able to follow the serving code from the moment a request arrives to the moment a token leaves the GPU. During optimization and debugging, we can inspect the implementation and change it where the work requires.

The routing layer grew out of necessity. Requests need to land on the GPU that already holds the right KV context. That means radix-tree lookup across the cluster, cost-function dispatch, session affinity and backpressure. We put that coordination in the routing binary, without an external coordination service such as etcd or NATS.

For a single GPU, the compute binary serves HTTP on its own. One process, one card. For clusters, the two binaries talk over ORPC — our binary RPC protocol over QUIC.

For supported decode paths, CUDA graphs let the engine capture GPU work and replay it on later iterations. This reduces repeated CPU work launching individual kernels.

The code around the GPU

The model’s forward pass is only part of a running service. Requests still need transport, routing, backpressure and a response path.

Our shared Rust foundation supplies HTTP/1.1, HTTP/2 and HTTP/3, TLS termination, QUIC transport and io_uring-based I/O. The engine and router share serving code rather than maintaining separate HTTP implementations.

A hardware abstraction layer separates device-level work from the model implementation. The serving layer, scheduler and GPU code remain distinct parts of the system, each with a defined job.

These choices shape how the engine can be deployed, investigated and maintained.

What it costs

Writing the engine also makes model support our responsibility.

When a new model architecture lands, supporting it can mean implementing an attention variant, positional encoding, sampling behavior or memory layout. Then we have to evaluate that implementation. We cannot inherit the result simply by upgrading an inference dependency.

That is a substantial maintenance commitment. We accepted it because control over GPU execution, routing and serving lets us make engineering decisions across those layers. It also leaves the work of keeping them correct with us.

Where the engine fits

The inference engine serves models within the larger Origon system. Agent orchestration directs the work, AI Datastore supplies knowledge and memory, and applications connect the result to people and business tools.

Explore how these components fit together through AI Engineering.