The tenant fallback that served one company's books to another

Multi-tenant bugs are rarely dramatic. Nothing throws, nothing pages, no dashboard turns red. A workspace opens its ledger and there are numbers in it, and the numbers belong to somebody else.

Here is one we found on a finance platform that connects each client workspace to its own ERP — Odoo or Zoho — and lets an AI agent read and post real invoices against it. The system was built fail-closed on purpose, and it still did this.

The shape of the bug

Each tenant resolves its own ERP connection. The resolver returns one of a few things: a connection, ErrNoPrimary when several are configured and none is marked primary, an unseal failure when the stored credentials cannot be decrypted, or a plain not-found when the tenant has connected nothing at all.

Three of those four branches refused the request. The reasoning was written into the code, and it was correct:

falling back to a global ERP could post their documents to the wrong system

The fourth branch — not-found — fell through to the process-wide, environment-configured adapter.

That adapter is not a neutral default. On a shared deployment it is the operator's own ERP: the books belonging to the company running the platform. So any workspace that had connected nothing could read that ledger through chat and through every document surface. And because the write path used the same resolution, a workspace confirming a document posted it into the operator's books.

We found it on staging, where two workspaces with no connection of their own were already transacting against the operator's Odoo.

Why it survived review

Read portFor and the bug is not obvious, because the function is mostly correct. Four branches, three of them refusing, one returning a fallback that in single-tenant development is exactly the right thing. The env-configured adapter is how the thing runs on a laptop. Nobody wrote a fallback for multi-tenant production; they wrote a fallback for local development and it outlived the assumption.

But that is not the interesting part. The interesting part is the test.

There was a unit test named TestUnnamedSystemStillUsesPrimary. Read the name: it asserts that when the caller does not name a system, routing still goes to the tenant's primary connection. That is precisely the property whose absence caused the leak, and the test was green.

It was green because it was built on an empty resolver. With no connections configured, the resolver returned not-found, the code fell through to the global adapter, and the test asserted it got the adapter it expected. The assertion passed. What it actually pinned was the fallback.

So the codebase contained a test that, by name, promised tenant-scoped routing, and by behaviour guaranteed the global fallback. Anybody who later removed the fallback would have seen that test go red and concluded they had broken primary routing. The test was not failing to catch the bug. It was defending it.

The fix

Refuse, and make the escape hatch loud and opt-in:

port, err := r.ResolvePrimary(ctx, tenant)
switch {
case err == nil:
    return port, nil
case errors.Is(err, tenantport.ErrNotFound):
    if !cfg.GlobalFallback {
        // The env-configured ERP is the operator's own books. A tenant that
        // has connected nothing gets told to connect one, not handed ours.
        return nil, tenantport.ErrNotConnected
    }
    log.Warn("serving the operator's ERP to an unconnected tenant",
        "tenant", tenant, "books", cfg.GlobalERPName)
    return globalPort, nil
default:
    return nil, err // ErrNoPrimary, unseal failures: refuse, as before
}

Four decisions in there are worth naming, because they generalise past this bug.

Opt-in, not opt-out. DAIRA_ERP_GLOBAL_FALLBACK defaults to off. Deployments that genuinely serve one workspace can switch it on. The direction matters: the failure mode is silent and severe, so the safe state has to be the one you get by doing nothing. An opt-out flag protects only the deployments whose operator already knew about the problem.

The escape hatch logs what it is about to share. If you must keep a dangerous path, make it announce itself by name. A warning naming the books being shared is the difference between a documented trade-off and a trap for the next operator.

ErrNotConnected carries the user's next action. The message is "connect one under Settings → Integrations". A refusal that leaves the user unable to proceed gets escalated, and escalations get resolved by someone turning the fallback back on.

Consistency was the actual argument. The surrounding code already refused in three analogous cases with a stated rationale. The fix was not new policy, it was applying the existing one to the branch that had escaped it. That is a much easier review than a novel security argument, and it is usually how these land.

The test, rewritten

Two new tests, and one deletion that mattered more than either:

// Covers read AND write. The write path is what posted documents into the
// wrong books, and it was not previously asserted at all.
func TestUnconnectedTenantIsRefusedNotGivenTheGlobalERP(t *testing.T)

// Pins the escape hatch, so removing it is a deliberate act.
func TestUnconnectedTenantUsesFallbackWhenExplicitlyOptedIn(t *testing.T)

And TestUnnamedSystemStillUsesPrimary was rewritten against a tenant that actually has a connection, so it now tests what its name always claimed.

That is the transferable lesson, and it is not about multi-tenancy. A test built on an empty fixture asserts whatever the fallback does. It is one of the most reliable ways to produce a green suite that guarantees the wrong behaviour, because the name describes the intent and the fixture quietly removes the conditions the intent applies to. When you read a test, read what it constructs before you read what it asserts. If the setup is empty, the assertion is about your default path, whatever the function is called.

The takeaway

A system that fails closed in most places does not fail closed. Isolation is a property of every branch, and it only takes the one written for a laptop.

Two things are worth doing in any multi-tenant codebase this week. Enumerate every place a tenant-scoped lookup can miss, and check what each one returns — not what it is documented to return. Then look at the tests covering those paths and ask what fixture they are built on. The leak is usually not in the code somebody reviewed carefully. It is in the branch that only development ever exercised, held in place by a test whose name says otherwise.