Postgres holds the state, the queue holds the delivery

Every backend reaches the point where something has to outlive the request. On a document-processing platform we were building, it arrived as OCR plus an LLM round-trip running inside the HTTP handler — so a burst of uploads degraded interactive chat latency on the same process. Alongside it were three unsupervised go func() calls with no retries, no backpressure, and work silently lost on every restart.

The fix is a queue and a worker, which everybody knows. The part that decides whether you get a reliable system or a new category of incident is a single question: what is the system of record?

If the answer is the queue, you will spend the next year debugging it.

The split

Postgres holds state. The queue holds delivery. Nothing else.

CREATE TABLE jobs (
  id          uuid PRIMARY KEY,
  tenant_id   text NOT NULL,
  kind        text NOT NULL,
  state       text NOT NULL,   -- queued | claimed | done | failed
  attempts    int  NOT NULL DEFAULT 0,
  run_after   timestamptz NOT NULL DEFAULT now(),
  claimed_by  text,
  lease_until timestamptz,
  payload     jsonb NOT NULL
);
ALTER TABLE jobs ENABLE ROW LEVEL SECURITY;
ALTER TABLE jobs FORCE ROW LEVEL SECURITY;

A queue message carries {job_id, tenant_id} and nothing else.

That constraint is where all the benefits come from, so it is worth being pedantic about. The message is not a work item. It is a doorbell.

The job row is the outbox

The row is written inside the caller's transaction. The publish happens after commit.

If the publish fails, the row is still there in queued, and a sweeper republishes it. So the table is the outbox — there is no separate outbox table, no change-data-capture relay, no second thing to operate. The classic dual-write problem (commit the transaction, fail to publish, lose the work) disappears, not because we solved it but because the durable record never depended on the publish succeeding.

This is the single highest-leverage decision in the design and it costs one sweeper.

At-least-once delivery becomes harmless

Every queue will deliver twice eventually. Usually that is a problem you handle by making handlers idempotent, which is real work and easy to get subtly wrong.

Here it is handled by the claim:

UPDATE jobs
   SET state = 'claimed', claimed_by = $worker,
       lease_until = now() + interval '5 minutes', attempts = attempts + 1
 WHERE id = $job_id
   AND state = 'queued'
   AND run_after <= now()
RETURNING payload;

A duplicate message loses the race, gets zero rows, and drops itself. A message for a job that already finished finds state = 'done', also gets zero rows, and drops itself. The handler never runs twice because the database decides who runs, and the database is the one participant with a consistent view.

Because the message carries no payload, handlers re-read authoritative state under the tenant clamp. A redelivered message cannot carry stale data, because it carries no data.

The queue holds nothing worth protecting

We used SQS behind a narrow interface. Replacing it later is an adapter change with no migration and no data to move, because there is nothing in it but doorbells. That is not a hypothetical benefit — it is what lets you start on SQS, or Redis, or a Postgres LISTEN/NOTIFY, without that choice becoming architecture.

The worker is a new door into your tenancy model

This is the part to review hardest, and it is the part most write-ups omit.

Every tenant-isolation guarantee in an HTTP service is anchored to the request: middleware reads the session, establishes the tenant, and everything downstream inherits it. A worker has no request. It has a message.

So the tenant comes off the message and the clamp is established before any job data is touched. A worker that forgets this is a cross-tenant leak with no HTTP request to blame and no user to correlate it to — which makes it both more likely and much harder to find than the equivalent bug in a handler.

Two details we only learned by testing this against a real Postgres.

Dedup must collapse within a tenant and not across tenants. The partial unique index that makes enqueue idempotent has to include tenant_id, or two tenants uploading the same document deduplicate into one job and one of them silently never gets their result.

A tenantless read returns zero rows, not an error — and which one you get depends on your column type. This is the subtlety worth the whole section. Once set_config has touched a GUC it stays defined but empty after commit, so current_setting stops raising. On tables whose tenant_id is a uuid, an empty setting still fails loudly, because casting '' to uuid raises 22P02. On tables using TEXT — ours, matching two existing tables — an empty tenant simply matches nothing and returns zero rows.

Zero rows is the correct behaviour: the defect you are defending against is returning every row. But it means your fail-closed test suite cannot assume an exception. If you assert "a tenantless read raises" you will have a green test on your uuid tables and no coverage at all on your TEXT ones. Assert the row count. The write side, separately, is still a hard rejection via WITH CHECK.

Queue depth is not your backlog

A consequence of the split that catches people operationally: because Postgres holds state and the queue holds delivery, queue depth tells you almost nothing.

A job can be queued in the table with no message in flight, waiting for the sweeper. A message can be in flight for a job already done. Depth measures doorbells, and you care about work.

Alert on the table:

-- real backlog
SELECT count(*) FROM jobs WHERE state = 'queued' AND run_after <= now();
-- stuck: leases that expired without an ack
SELECT count(*) FROM jobs WHERE state = 'claimed' AND lease_until < now();

Every dashboard we have inherited from someone else graphed queue depth. It is the number the queue gives you for free, and it is the wrong one.

Two decisions that are less obvious

One binary, two roles. DAIRA_ROLE=web|worker|all — so the tiers scale independently without a second image or a second deploy pipeline. The cost of a separate worker image is not the Dockerfile, it is the permanent risk of the two drifting.

Production refuses to boot on the in-memory implementations. Dev runs without AWS, so there are in-memory store and queue adapters. Production will not start on them. A queue that discards accepted work on restart is not a degraded mode, it is a correctness stub, and the difference between "slower" and "silently loses your customers' uploads" is exactly the distinction a startup check exists to enforce.

One thing we flagged rather than fixed: the sweeper iterates every tenant and issues one clamped query each, so a sweep is O(N) round trips. That is deliberate — it keeps recovery inside the forced-RLS path instead of opening a cross-tenant read — and it is fine at hundreds of tenants on a 60-second interval. The code says so, and says what to do when it stops being fine.

The takeaway

Put the state in the database, inside the transaction that created the work. Let the queue carry an id and a tenant and nothing else. Then at-least-once delivery, the dual-write problem, duplicate handling and queue-vendor lock-in all stop being separate problems you solve individually, because none of them can affect the durable record.

And if you are adding a worker to a multi-tenant system: the worker is a new entry point into your isolation model, with no request to hang it off. Whatever your handlers do to establish the tenant, the worker has to do explicitly, before it touches anything — and your fail-closed tests need to assert row counts rather than exceptions, or they will pass without covering half your tables.