← All notes

A queue without a lock

A few days ago I wrote that Guardian Bots should stay on Postgres because its job queue depended on it. That advice got overruled, the migration happened, and it turned out fine. Worth explaining what the queue relied on and what it relies on now.

What SKIP LOCKED did

In Postgres, a worker claiming the next job runs something like SELECT ... FOR UPDATE SKIP LOCKED. It locks the row it is about to take, and any other worker running the same query skips rows that are already locked and takes the next free one. Two workers never get the same job, and neither has to wait for the other. It is one statement and the database does all the coordination.

That is a good guarantee, and it is specific to Postgres. MongoDB has no row locks to skip.

What replaced it

One atomic update:

const run = await db.runs.findOneAndUpdate(
  { status: "queued" },
  { $set: { status: "running", startedAt: new Date(), workerId } },
  { sort: { queuedAt: 1 }, returnDocument: "after" },
);

Find the oldest queued run and flip it to running in a single operation. MongoDB guarantees that a findOneAndUpdate on one document is atomic. Two workers race for the same run; the database applies one update first; the second worker's filter no longer matches that document, because its status is no longer queued, so the second worker gets the next one instead.

Same outcome as SKIP LOCKED. No transaction, no lock, no retry loop. The claim is the update, and the update either happens to you or it does not.

Claims are issued one at a time up to a concurrency limit, so a worker with four free slots makes four separate claims rather than one claim for four. That keeps each claim atomic and keeps the effective behavior identical to the Postgres version.

The other things Postgres was doing for free

A migration like this is mostly about noticing what the old database was doing silently.

Foreign keys. Postgres deleted a bot's runs and alerts automatically when the bot was deleted. MongoDB does not have that, so the delete now removes them explicitly, and runs carry a copy of their owner's id so per-user queries do not need a join.

Magic links. The old code marked a link consumed with an update and relied on the transaction to make that safe. The new code consumes the link with the same findOneAndUpdate pattern as the queue: filter on unconsumed, set to consumed, and a link clicked twice signs in exactly once. Expiry is a TTL index, so the database deletes stale links on its own.

Migrations. There are no SQL migrations any more. Mongo creates collections on first write, so the startup step now creates indexes instead, which is idempotent and safe on every boot.

The test that earns the migration

The repository ships a db:verify command that takes a connection string and proves it end to end: connect, ping, create an index, write, read, delete. One of the steps exists specifically for the queue:

const claimed = await db.users.findOneAndUpdate(
  { _id: probeId, name: "probe updated" },
  { $set: { name: "probe claimed" } },
);
const second = await db.users.findOneAndUpdate(
  { _id: probeId, name: "probe updated" },
  { $set: { name: "probe claimed twice" } },
);
if (second) throw new Error("the same document was claimed twice");

It claims a probe document, then tries to claim it again with the same filter, and fails loudly if the second attempt succeeds. That is the entire guarantee the queue rests on, reduced to something you can run against a real database in a second. CI runs it against a real MongoDB on every push.

If that step ever fails, the queue would double-run bots, and you would know before deploying rather than after.

What I got wrong

I said the queue was a redesign, not a translation. It was closer to a translation than I expected: the atomic-update pattern is the Mongo idiom for exactly this, and it is shorter than the SQL it replaced. The part that was real work was everything Postgres had been doing implicitly: cascades, transactions around link consumption, migrations. None of it was hard. All of it had to be found.

The migration was the right call. The caution was still worth having, because the things that needed finding were found by looking for them.