Background job queue with retries

A job queue on Postgres with retries, backoff, scheduling and a small dashboard.

Why build it

Queues are in almost every production system. Building one teaches locking, idempotency and failure handling at the root.

Features

• Enqueue with payload and priority
• Workers claim jobs with SKIP LOCKED
• Retries with exponential backoff
• Delayed and recurring jobs
• Dashboard of queued, running and failed jobs

Roadmap

1. p1

   Core queue

   • Jobs table and state machine
   • Claim with SELECT ... FOR UPDATE SKIP LOCKED
   • Worker loop with graceful shutdown
2. p2

   Reliability

   • Retry policy and dead-letter state
   • Visibility timeout for crashed workers
   • Idempotency keys
3. p3

   Scheduling

   • Delayed jobs
   • Cron-style recurring jobs without duplicates
4. p4

   Observe

   • Dashboard with failure details
   • Metrics: throughput and latency
   • Chaos test: kill workers mid-job

Common pitfalls

• Marking a job done before the work is committed
• Retrying non-idempotent jobs and charging a customer twice

Skills you will practise

• Postgres locking
• Idempotency
• Failure testing
• Observability