OpenMake
01
The code was tied to one server's dialect
Until then local inference went through Ollama's own HTTP API: line-delimited JSON streaming, a rotating pool of keys, and separate dedicated routes for chat, embeddings, and web search. That shape had worked its way deep into the client code.
The destination was vLLM behind a LiteLLM proxy, which speaks only the OpenAI-compatible protocol. This was not a matter of changing an address. The streaming format differs, usage is counted in a different unit, and some capabilities exist on only one side.
And more than a hundred places called this code. Fixing all of them while simultaneously changing the protocol would have made it impossible to tell what broke.
02
The compatible signature came first
So the first step was not the migration but the preparation. A new module went in as a thin wrapper over the OpenAI SDK, but its outward signature was kept identical to the existing client. From a caller's point of view it was the same object under a different name.
The 685 lines of type definitions deliberately kept the old names. Renaming things and replacing a backend in the same commit means that when something breaks later you cannot tell which change caused it. Compatibility first, cleanup deferred.
- The stream parser was rewritten for the OpenAI delta format and its reasoning field
- The reasoning-effort setting went in as an opt-in and defaults to off
- Usage tracking moved from a time basis to a token basis — a change that gives a limit of the same name a different meaning, so it was called out separately
- The multi-turn tool-calling loop was rebuilt against the OpenAI tool specification
03
Changing the backend meant losing a capability
The old backend had a dedicated web-search route. vLLM does not. Standardizing the protocol also removes whatever the standard does not carry.
It could not simply disappear, so it had to move to another layer. An adapter now delegates web-search calls to the MCP tool side. A capability the inference server used to provide became the tool layer's responsibility.
As it turns out that is the better home for it. Search is a tool's job rather than an inference engine's, and changing the search provider later no longer touches the inference backend.
04
Moving 100 call sites at once, deleting 4,500 lines
The second step was mechanical. A codemod ran across the repository rewriting import paths and renaming configuration references. Because the signature already matched, callers compiled as soon as their path was updated.
Then the old directory was deleted outright — client, key manager, key pool, cooldown, stream parser, connection pool, interceptors: 15 files, roughly 4,500 lines. Tests that depended on the retired implementation went with them.
The deletion is the point. Ship the new implementation and keep the old one and you have two live paths, with fewer and fewer people knowing which is real. The compatible signature was a bridge for the migration, not a device for holding two implementations together indefinitely.
05
Configuration, labels, and one thing left on purpose
The third step was the part people see. The example configuration file shed a pile of backend-specific variables and was reorganized around eight new ones. Multiple keys, multiple model slots, and a cluster node list all disappeared here, because a single proxy address was now enough.
One thing was deliberately kept. The string identifying this provider internally was baked into a database constraint and into model identifier prefixes, so renaming it would have dragged a data migration along. It was not a change that had to happen right then, so it was written down as follow-up work and left in place.
Deciding to leave something is different from forgetting it. This one was actually cleared thirteen days later.
06
That same evening, what the codemod missed
A bulk rewrite only catches what matches the pattern. By that evening the leftovers had surfaced.
References that went through the configuration object were caught, but places reading the environment variable directly had a different shape and survived untouched. A default model name was also hardcoded in one place; it became a lookup that re-reads the environment each time, so a restart or a test override takes effect immediately.
The most visible one was the code that checks the model list. It called the old backend's proprietary route, assumed that route's response shape, and on failure suggested the old backend's command. It moved to the OpenAI standard route and response shape, and the error message now points at the proxy configuration and serving options instead — printing which models are currently exposed so the cause is obvious.
Validation logic for a cloud suffix that only existed on the old backend also went away here. The new backend has no such distinction.
07
Once the boundary was standard, the far side got easy
What happened after the migration is what shows its value. Two days later, per-model serving scripts, service units, and proxy alias configuration went in. Five days later a local model catalog appeared, handling four chat models and one embedding model as a list that an environment variable can replace wholesale.
Over the following two weeks the actual served model lineup changed several times. Application code was not touched once. The proxy routes by model name and the client only needs to know a single address, so every change ended inside a configuration file.
Finally, at the end of May, the internal provider identifier was renamed to a standard one and the migration was closed out. That is exactly the debt left on purpose in step three.
08
The compatibility layer's invoice
The compatible signature clearly worked. Moving more than a hundred call sites in a day is what it bought, and without it the backend swap and the caller edits would have tangled into something with no diagnosable cause.
It also leaves an invoice. Type definitions still carrying old names, direct references the pattern missed, an identifier pinned by a database constraint — that is the invoice. The migration itself took a day; paying the invoice off took thirteen.
What made the difference was deciding to pay it. Every deliberate leftover had its reason and its intended cleanup written into the commit, so the residue got erased one item at a time instead of being forgotten. A compatibility layer is useful as a bridge; left standing after you cross it, it is just a second implementation.
Source evidence