Building a Task Queue From Scratch

I wanted one thing: a Node.js service and a Python service pulling work from the same queue. Not two queues sitting next to each other, one queue, genuinely shared, so either language could pick up any job.

The obvious move is BullMQ on the Node side and Celery on the Python side, both backed by Redis. That's when I ran into the problem that ended up shaping this entire project: BullMQ and Celery don't actually share anything. Each one writes its own private data format into Redis. A BullMQ producer enqueues a job in a shape only BullMQ understands. A Celery worker looking at that same data sees nothing usable. They both happen to use Redis, the way two people can both happen to use the same filing cabinet while writing in languages neither one can read.

So I built the queue myself, directly on Redis's basic data structures, with one written contract both languages implement independently. No RPC between the two services, no shared library across the language boundary, just an agreed-upon shape for the data sitting in Redis.

The contract that makes this work

Before writing any code that actually does something, I wrote down exactly what a job looks like as data: its fields, its possible states, and which Redis keys represent the queue. This became the one thing both languages had to agree on, character for character, since they're reading and writing the exact same bytes in Redis, not talking to each other through any API.

A job has four states, not more: pending (waiting), active (a worker has it), completed, dead (given up on). A retry isn't a fifth state. It's a job going back to pending with a delayed availability ; availableAt and one more attempt counted.

export interface Job<TPayload = unknown> {
  id: string;
  type: string;
  payload: TPayload;
  priority: number;
  status: "pending" | "active" | "completed" | "dead";
  attempts: number;
  maxAttempts: number;
  availableAt: number;      // not claimable until now >= this
  leaseExpiresAt: number | null;
  result: unknown | null;
  error: string | null;
  idempotencyKey: string | null;
}

The Python side reimplements this as a dataclass with the same fields, snake_case instead of camelCase, since that's idiomatic Python. That mismatch mattered more than I expected: the wire format actually sitting in Redis is camelCase (whatever the TypeScript side writes), so the Python side needs an explicit translation function at the boundary rather than hoping the two naming conventions happen to line up. Small detail, but it's exactly the kind of thing that only shows up once two independent implementations actually try to read each other's data for real.

Sorting by priority and delay at the same time

Jobs live in a Redis sorted set, queue:pending, where Redis keeps every member ordered by a numeric score automatically. The obvious score would just be the timestamp: oldest job first. But I needed two things sorted at once, priority (urgent jobs first) and delayed availability (a job retrying after a failure shouldn't be visible until its backoff period ends).

The fix is one formula:

score = priority * 10_000_000_000_000 + availableAtMs

A millisecond timestamp never reaches 13 digits until the year 2286, so multiplying priority by something bigger than any possible timestamp guarantees priority always wins the comparison first. The timestamp only breaks ties between jobs of the same priority. One structure, one query, gives you priority ordering, FIFO within a priority tier, and delayed availability, all at once.

Claiming a job without a race condition

Here's the part that actually needs care. Two workers, in two different languages, could both look at the queue at nearly the same instant and both decide to take the same job. Whichever fix I used had to make claiming atomic: either a worker gets the job entirely, or it doesn't get it at all, with no window in between where two workers both think they succeeded.

Redis solves this with Lua scripting. A script submitted to Redis runs as one uninterruptible unit, nothing else can execute in the middle of it. I used the claim code as a single .lua script, loaded by both the Node.js and Python clients, not two separate implementations that are supposed to behave identically, the literal same file:

local job_ids = redis.call('ZRANGE', KEYS[1], 0, 0)
if #job_ids == 0 then
  return nil
end

local job_id = job_ids[1]
redis.call('ZREM', KEYS[1], job_id)                    -- remove from pending
redis.call('ZADD', KEYS[2], lease_expires_at, job_id)   -- add to active, with a lease

local job = cjson.decode(redis.call('HGET', 'job:' .. job_id, 'data'))
job.status = 'active'
job.attempts = job.attempts + 1
redis.call('HSET', 'job:' .. job_id, 'data', cjson.encode(job))
return cjson.encode(job)

Both language clients call this exact script through Redis's EVAL command. Neither one has its own idea of what "claiming" means. There's exactly one definition, and it lives in Redis, not in either codebase.

What happens when a worker just dies

A claimed job doesn't just belong to a worker forever. It gets a lease, a deadline (leaseExpiresAt, 30 seconds out by default), like a parking meter. If the worker finishes before the meter runs out, the lease stops mattering. If the worker crashes and the meter runs out with nobody reporting back, a separate process called the reaper notices and puts the job back into circulation, exactly as if it had failed normally.

This wasn't a hypothetical I tested in isolation. During development, a worker genuinely crashed mid-job because of an unrelated bug, leaving a job stranded with an active lease and no one home. When I built the reaper afterward, it found that real, several-hours-old orphaned job and correctly recovered it on the next scan. Nothing staged about that, it was just sitting there.

Retrying, backing off, and giving up

When a job fails, whether the handler threw an exception or the reaper found it abandoned, the same function decides what happens next. If it's out of attempts, it moves to a dead letter set for a human to look at later. Otherwise it goes back to pending, but not immediately: the wait grows exponentially with each failure (roughly 1s, 2s, 4s, capped), plus a little random jitter so a burst of simultaneous failures doesn't cause a synchronized retry storm.

def _compute_backoff_ms(attempts: int) -> int:
    backoff = min(BASE_BACKOFF_MS * (2 ** (attempts - 1)), MAX_BACKOFF_MS)
    jitter = random.uniform(0, backoff * 0.1)
    return int(backoff + jitter)

The Lua script that actually moves a job between queues stays deliberately dumb, it doesn't know what a retry is or how backoff math works, it just moves a job from one sorted set to another with whatever data it's handed. All the deciding happens in application code. That split kept the one piece of logic that's genuinely hard to get right (atomic multi-key updates) small and boring, while the part that changes and has actual judgment calls in it (how long to wait, when to give up) stayed in a normal, readable, testable function.

There's a good honesty check worth naming here: the claim script above, as originally written, just grabbed the lowest-scored job unconditionally. That was correct right up until retries existed. Once a failed job could have a future availableAt, a delayed high-priority retry could still sort ahead of an available low-priority job (priority dominates the score, remember), and the naive script would have claimed it immediately, ignoring the fact that it wasn't due yet. The fix was to scan the lowest few candidates in order and claim the first one actually due, rather than blindly taking the minimum. I'd left a comment documenting this exact limitation when I first wrote the script, specifically so it wouldn't get forgotten, and it didn't.

Not every atomic operation needs a Lua script

Submitting a job twice by accident, a double click, a retried HTTP request, is a different problem, and it turned out to need much less machinery. An idempotency key only ever touches one Redis key, so Redis's own SET key value NX (set only if the key doesn't already exist) is already atomic on its own:

const claimed = await redis.set(dedupKey, jobId, "EX", DEDUP_TTL_SECONDS, "NX");
if (claimed !== "OK") {
  // someone already claimed this key, return their job instead of creating a new one
}

Worth noticing as a general lesson: claiming and failing a job need a Lua script because they touch multiple keys as one operation. Deduplication only touches one key, so a single conditional command is enough. Reach for the heavier tool only when the problem actually needs it.

Proving it's actually shared, not just two workers that happen to coexist

The real test was running both workers against the same queue at once and checking, rigorously, whether any job ever got claimed by both. I diffed the claimed-job-id logs from each worker after a mixed run: zero overlap, every time.

The more interesting result was one I hadn't specifically engineered: retries don't stay with the worker that first touched them. A job that failed on its first attempt in Python got picked up and completed on its retry by Node.js, and separately, a job that failed in Node.js had its next attempt claimed by Python. Neither worker knows the other exists. They just both speak the same protocol against the same data, and whichever one is free next gets whatever's next in line.

What this is actually good for

This shape of system, thin API accepts and persists, a queue holds work, workers claim and process, isn't specific to any one kind of job. The same pattern shows up behind webhook delivery (retry a flaky endpoint with backoff, dead-letter it if it never recovers), transactional email, report generation, image processing, or slow AI inference behind a "submit and poll" API. The interesting part was never the concept of a queue. It was making the atomicity actually hold up once two independent, mutually unaware programs are reading and writing the same data at the same time, and being honest about the assumptions (like "nothing has a future availableAt yet") that stop being true as the system grows, catching them, and fixing them properly instead of working around them.