Six months serving vLLM on a DGX Spark: what one memory pool teaches you

01

One pool, three services

We run the team's stack on two machines. A Mac mini runs the application: API and web under PM2, with Postgres, Redis and the sandboxed tool processes in Docker. An NVIDIA DGX Spark GB10 sits next to it running vLLM behind a LiteLLM proxy, plus BGE-m3 for embeddings and FLUX for image generation. They talk over a private Tailscale link.

The GB10 puts CPU and GPU on one physical memory pool. 127.5 GB total, of which vLLM sees 121.63 GiB as device memory. nvidia-smi reports usage as N/A — it has nothing separate to measure. So the budget is not a GPU number you read off a tool; it is a number you assign, and then verify from boot logs.

• vllm-chat (qwen3.8-27b-fp8, 262K) — budget 0.47, filling 57.17 GiB: weights 28.95, CUDA graphs 1.2, non-torch overhead about 3, leaving 20.18 GiB of KV cache = 587,202 tokens at fp8. At a full 262K request that is 2.24 concurrent.
• vllm-embed (bge-m3) — budget 0.05, about 6.1 GiB: weights 1.06, remainder KV.
• flux (FLUX.2-klein-9B) — ratio unspecified, 32.5 GiB resident after load.
• Total about 95.8 GiB, leaving roughly 26 GiB for the OS, Docker and processes. MemAvailable sits at 15–16 GB.

02

What that budget actually says

Two things fall out of those numbers. FLUX is the most expensive tenant we have and it is not even the product — it holds 32.5 GiB resident so that an occasional image request does not pay a cold load. And the chat model's fraction is the only real dial: raising it takes memory directly from the OS, on a box where the OS and the model are the same pool.

The split between the two machines is deliberate. Inference and application have different resource profiles, and co-locating them means the model evicts Postgres from page cache at the worst possible moment. But the split is also the source of every failure below. Each one has the same shape: a fact that lives on one side of the link and is assumed on the other.

03

The alias was doing its job

On September 2, during a routine check of the DGX, I read env/chat.env and the root field of vLLM's /v1/models. The model actually being served was qwen3.8-27b-fp8. The application had been calling it qwen3.6-35b-a3b since August 31, when we swapped it.

Nothing had failed. --served-model-name carried the old name as an alias alongside the new one, which is exactly what you do during a migration so the application does not break mid-swap. It worked. The app kept requesting the old name, LiteLLM kept resolving it, vLLM kept answering, and the model list in the UI kept displaying a model that had not existed for two days. The convenience that was supposed to protect the migration is what hid its completion.

Two signals were visible in that window, and we read both as something else. Decode throughput had dropped from 58 tok/s to 10.7 tok/s — a 5× slowdown that matched our own August 15 benchmark of Qwen3.8 exactly, and it looked like load variance. And reasoning_effort=high started returning 400, which looked like a provider quirk. Neither pointed at a model change, because the name in every log line was the name we expected. The regression stayed unattributed for two days.

The lesson we wrote down: a local model's identity is what /v1/models says its root is, not what a LiteLLM alias resolves. An alias is a routing convenience; it is not a claim about what is running. So boot and periodic probes now read the gateway's /model/info and refresh the catalog from what is actually there, and the static list survives only as a fallback for when the probe itself fails.

04

Fan-out meets real rate limits

Discussion and Deep Research fan out to several expert calls at once. Against free and developer keys that is a burst, and providers treat it as one: five parallel calls came back 5 of 5 with 429 on one key, and 3 of 5 on another.

Three separate things were wrong, and they had to be fixed as three changes.

• The external execution client had no notion of a per-provider budget. It now sits behind a per-provider semaphore with exponential backoff on 429, and a follow-up review made it honor Retry-After and stop misreading unrelated 429s as our own.
• Deep Research chose its fan-out width without asking what the provider could take. Concurrency and timeouts now follow the provider hint.
• The SDK was retrying on its own. A long-reasoning model was being cut off every 360 seconds by a retry we never asked for. SDK retries are now zero, with a multiplier on its timeout.

05

The change I would defend least confidently

In the same release, an explicitly chosen external model that cannot run now surfaces an error instead of silently falling back to local.

Silent fallback keeps the request alive, which is what you want at 3am. It also means a user who picked a specific model gets an answer from a different one and is never told — which is the alias problem again, in a different costume. We decided the second cost is higher, but I would not call it obvious.

06

Overflow handled rather than hidden

We serve a 262K window and still hit it. Long agent runs accumulate tool output, and images are not free. The safety net has three stages, and the order encodes what we were willing to lose: truncate the input first, keeping the system message and the most recent turns; then reduce max_tokens down to a guaranteed floor; and only if the system message alone still exceeds the window does a ContextOverflowError return HTTP 413 with an audit record and an automatic webhook alert.

Output is protected first. A reply truncated mid-sentence is worse than one that had less context to work from, because the user can see the second failure and act on it. Truncation also anchors the first user message: on an agent task that message is the goal, and losing it produces a run that is busy and pointless.

The threshold is not a constant either. It resolves in order — explicit env override, then the boot-time probe, then the catalog, then the default — because the model on the other side of the link is free to change, as the alias incident found out the hard way. This one is older than the rest; it shipped on May 25, right after the vLLM migration.

07

A cap enforced in one process, assumed in another

Per-request prompt images are now capped at vLLM's own --limit-mm-per-prompt value of 8. The cap already existed on the serving side. It did not exist in the application, so the application would happily assemble a ninth image and let the server reject the whole request.

Same shape as the alias incident, and by now I recognize it: a constraint enforced in one process and transcribed into another. The only defense we have found is to read the constraint from its source at runtime instead of copying it. A related fix landed the next release — assistant reasoning is preserved into the next turn of the local tool loop, and vLLM tool_call ids are preserved, because multi-turn tool use depends on both.

08

It is not memory, and it is not users

The intuitive limit on a box like this is KV cache. For us it never has been.

• KV has room. 20.18 GiB is 587,202 tokens — only 2.24 concurrent requests at a full 262K, but real turns are 12–17K, so it holds 30–40 sequences. At about 1,100 requests a day from 2 active users, KV has not been the constraint once.
• Prefill is compute-bound, roughly 800–900 tok/s. A 17K first turn takes 15–22 seconds on a cache miss, 5 seconds on a hit. We raised max-num-batched-tokens from 8192 to 16384 and TTFT did not move at all.
• Decode is memory-bandwidth-bound at 10.7 tok/s single-stream. A dense 27B at FP8 means reading 29 GiB of weights for every token, so GB10's bandwidth is the ceiling. Raising MTP to 2–3 bought 4–5%.
• Concurrent users are a derived problem, not a primary one. When users increase, what fills up is the prefill queue, not KV. One 17K prompt occupies the GPU for about 20 seconds; N people sending a first turn at once simply stand in line.

09

The levers, in the order we would pull them

First, reduce prefill itself — prefix cache hit rate and prompt diet, already down 27%. Second, a second inference node; the LiteLLM routing and cluster code already exists. Third, a smaller model or a return to MoE, worth about 5× on decode.

Buying more GPU memory is only interesting if we drop FLUX to grow KV, and that does not touch the throughput ceiling at all. Which is the useful thing to know before spending money: on this box, the number that looks like the limit is not the limit.

10

Would we buy another one

Yes. And the shape of that answer is the honest summary of six months.

Nothing above is a complaint about the hardware. All four failures were ours, on the application side of the link — a name transcribed instead of read, a burst sent without asking what the far end could take, an overflow path we had to stop hiding, a cap enforced in one process and assumed in another. The box does what we bought it for.

But scaling means a second Spark, not a dial, and that is the trade self-hosting actually makes. An API bill is operating expense that tracks usage. A box is capital expense that has to clear before the capacity exists. We will add one when the budget does — which is a different sentence from “we can scale when we need to,” and it is the true one.

The levers above are not a roadmap being announced. The first one is already 27% in, the routing and cluster code for the second exists, and the measurements in this post are three days old at the time of writing. We publish a log every week of what shipped and what did not, including the weeks that went badly; this post is the six-month version of that same habit. The next one will have different numbers, and probably a different bottleneck.

11

Who took these measurements

One thing is worth saying plainly, because it may change how you read everything above. This project is led by a non-developer. openmake_llm is built through what people now call vibe coding, with a professional developer supporting it.

The boot logs are real, the measurements are real, and the commits are public. But the person who took those measurements learned to take them while doing it. The prefill-versus-decode distinction in the bottleneck section was not knowledge brought to this project; it was something the project made unavoidable. The alias incident is in this post partly because it is a good story and partly because it is what learning by running a service in production actually looks like.

I mention it because the interesting question is no longer whether a model can write code. It is whether someone who could not have written this stack by hand can still operate it — measure it honestly, find where it lies to them, and fix the thing rather than the symptom. Six months in, the answer seems to be yes, with a great deal of measuring. The four failures above are the shape that cost takes.

So if you try it and the guide is wrong somewhere, please tell me. I would rather fix the documentation than answer the same question twice, and outside eyes on this codebase have already turned into commits more than once.