1.Approved with conditions: moving cron jobs to a queue
When it fits: Use this when the proposal is technical, the team mostly agrees on the problem, and the review is about whether the chosen solution is the cheapest one that works.
RFC-031: Run background jobs from a Postgres-backed queue instead of cron
Status: Approved Author: Nadia Farouk Reviewers: Grant Liu (database), Esther Mwangi (on-call), Pablo Ortega (billing) Approver: Ingrid Solberg Created: 2026-10-05 Comments close: 2026-10-16 Decision needed by: 2026-10-20, before Q4 planning locks Related: [[Background jobs]]
Summary
Our 23 cron jobs fail silently and cannot be retried. This RFC moves them to a job queue stored in the Postgres database we already run. Cost: about three engineer-weeks, no new infrastructure.
Background
Since June, 7 incidents traced back to cron jobs that failed without anyone noticing, including the invoice run on 2026-09-01 that sent 1,140 invoices a day late. Jobs run on one VM; if it restarts mid-run, the job is lost.
Goals
- Every job retries up to 3 times and alerts on final failure.
- A dashboard shows last run, duration and status for each job.
Non-goals
- Replacing Kafka for event processing.
- Sub-second scheduling.
Proposal
Use a queue library that stores jobs in a Postgres table and locks rows with SKIP LOCKED. Workers run in the existing app containers. Schedules move from crontab into code, reviewed like any other change.
Alternatives considered
Redis-based queue
- Mature, but adds a stateful service we would have to run and back up. Rejected for operational cost.
Do nothing
- Roughly one incident a month and four hours of on-call time each.
Risks and open questions
- Performance and cost: extra load on the primary database. Estimate under 2 percent CPU at current volume.
- Migration and rollback: migrate one job per day; crontab entries stay commented out for two weeks.
- Open question: does the invoice run need exactly-once behavior? (@Pablo)
Rollout plan
- Two low-risk jobs, week of 2026-10-26, exit criteria: two weeks with no missed runs.
- Remaining jobs, November, invoice run last.
Decision
- Outcome: Approved with conditions on 2026-10-19 by Ingrid Solberg
- Conditions: load test at 5x current job volume before the invoice run moves (Grant). Invoice job gets an idempotency key (Pablo's answer: yes, exactly-once matters).
- Follow-up: decision record DR-018; tickets JOB-201 to JOB-214.