10 min read

We built isolation, and in production it did nothing

We shipped OS-level isolation for external MCP servers in a day, then deleted it the next morning. The tool we had chosen turned out to quietly do nothing on the host we actually run on.

  • MCP
  • Sandbox
  • Security
  • Docker
OpenMake MCP catalog screen listing installable MCP servers
A server installed from the catalog ends up as a process on our host. The question is what that process can see.

SHIPPED / EVIDENCE

What shipped

External MCP stdio servers now run inside a non-root Docker container with every capability dropped. No host path is mounted, only explicitly configured environment variables are passed in, and the network is decided per server. Servers that cannot be isolated are marked as unisolated rather than hidden, and the server that executes arbitrary code was baked into the image so it could move into a container with no network at all.

Isolation from the first build
None
Time to rewrite
8 hours
Containerized
12 of 14
Arbitrary-code server network
none

01

It started as a leak, not as isolation

An external MCP server is installed from a catalog, approved by the user, and then spawned by our server as a child process. That spawn was handing the child the host's entire environment: database connection string, JWT signing key, token encryption key, LLM API key, everything.

The fix itself was small. The MCP SDK's stdio transport already provides a safe subset such as PATH and HOME as its base, so there was no reason for us to spread the full host environment on top of it. Only values a server explicitly configured are passed through now.

This is a deliberate behaviour change. A server that had been implicitly inheriting host values now has to name the keys it needs in its own configuration. We traded convenience for explicitness.

02

Policy picked the tool

Blocking the environment still leaves the process running on the host filesystem, able to read the repository and open secret files. Another layer was needed.

Here an existing policy got in the way: containers were for infrastructure only, so we avoided having the application spawn containers of its own. That pointed at bubblewrap, which isolates the filesystem, processes, and optionally the network without needing a daemon.

The implementation was written defensively. If the flag is off, if the host is not Linux, or if the binary is missing, it runs the original command and logs a warning. The judgement was that a missing sandbox should not take the service down. Eight unit tests and 52 MCP regression tests passed.

03

That defensive design is what made isolation zero

What surfaced the next morning is simple. Bubblewrap is Linux-only, and the production host is macOS — the service runs on a Mac mini.

So the safety valve we had written was firing every single time. Not Linux, therefore run the original command, log one warning, move on. The code existed, the tests passed, and the actual isolation was zero.

That is worse than a bug, because it was the design working as written. The tests verified that the gate opened and closed correctly; they never verified that in production the gate was always closed.

04

Rewritten in a day

A Docker Desktop container runs inside its Linux VM, so isolation genuinely works on any host with Docker, macOS included. The tool we picked to honour a policy was protecting nothing, so we changed the policy instead — restating its scope as isolating the external processes the application spawns, rather than containerizing the application itself.

The bubblewrap implementation was deleted along with its tests and design notes. Leaving a day-old parallel implementation behind only makes the next person wonder which one is real.

The new implementation wraps the command in a container run. Every capability is dropped, privilege escalation is blocked, it runs non-root, and process count and memory are capped. Not a single host path is mounted, so there is no route to a secret file at all. Environment variables reach the container only if they are in the server's own configuration.

  • The network is chosen per server, either a bridge or none at all
  • A configuration pointing at 127.0.0.1 inside the container is rewritten to the host address, so a server talking to a database on the host keeps working
  • One runtime image carries both node and uv, covering the npx-family and uvx-family servers together
  • Turning it on without the image built makes spawning fail, so it defaults to off

05

Turning it on, two things blocked it

Passing tests and actually booting are different problems, and this time we found the gap immediately.

First, the image build failed. The base image already ships a user at uid 1000, and we were trying to create another one at the same id. We reused the existing user instead of adding a new one.

Second, containers started but could not write to the package cache. A named volume is created owned by root, while the container runs non-root. Creating the cache directory inside the image already owned by that user means an empty volume inherits the ownership when it is first mounted.

With those two fixed, ten standard servers connected from containers, and the PostgreSQL server reached the database on the host through the address rewrite.

06

We did not hide the servers we could not isolate

Three servers were left over. Each depended on a binary installed directly on the host and would not run inside the generic runtime image.

There were two bad options: turn isolation on and let those servers die, or swallow the failure quietly and let them look isolated. Our earlier mistake was exactly the second kind, so we built a third option instead.

The per-server network policy column got one more value. On that value the server skips containers entirely and runs directly on the host, ignoring the isolation flag whether it is on or off. The unisolated state is written explicitly in configuration, and those servers never appear in the isolation log, so the difference is visible.

After a restart, 14 servers connected in production: 11 isolated in containers and three unisolated on the host. That pass also revealed that the fetch server had been pointing at an npm package that does not exist; corrected to the right one, it ran in a container fine.

07

The riskiest server went back inside first

One of the three left outside was the Python REPL, which executes arbitrary code. The most dangerous server had ended up the least protected, which was not something to leave alone.

The answer was to move the dependency from the host into the image. The package is installed at build time into a self-contained environment, kept separate from the cache path that gets a volume mounted at runtime. Because nothing has to be downloaded to run, it works even in a container with the network completely cut off.

In production configuration the server's command changed from a host absolute path to a name inside the container, and its network policy moved from unisolated to blocked. MCP initialization returned normally in the offline container, and after a restart the container distribution was 11 on a bridge and one with no network.

The server that runs arbitrary code now gets OS isolation and network blocking at the same time.

08

What a container cannot stop

A container limits what a process can see. It does not limit who can call the tool. On a service that is publicly reachable with open registration, that was the more urgent problem.

So a minimum-role gate was added per server. The arbitrary-code server is exposed only to admins and the browser automation server only to signed-in users. A guest who explicitly enabled that server and asked it to run code produced zero executions.

Every MCP tool call is written to the audit log. Guests have no user identifier, so that column is left null and the actor recorded in a separate field, which keeps the foreign key intact while still recording who called.

  • A CPU cap stops one server from taking over the host
  • Cache volumes are split per server so one server's package cache cannot contaminate another
  • Host address access was granted only to the single server that genuinely references 127.0.0.1
  • A read-only root filesystem is available as an opt-in

09

What we left, and what is left

One thing was deliberately not done: pinning package versions. Fixing versions for fourteen servers at once risks stopping all of them without knowing which one broke, so the supply chain risk was accepted knowingly for now. Splitting the caches per server at least cut the path contamination would travel.

Two servers still run unisolated on the host. Both depend on host-installed binaries rather than published packages, so the generic image does not cover them. A dedicated image is the remaining work, and until then the configuration and the logs both say unisolated.

The lesson that outlasted the isolation mechanism is not about bubblewrap or Docker. It is that a safety valve designed to fire quietly will fire quietly forever without anyone noticing. Isolated servers are now listed in the log, and the ones that are not say so in their configuration.

Source evidence

Source evidence

Back to Engineering Log