Markdown source

# RFC: Replace Background Job Queue with Priority Queue System

**Status:** Draft  
**Author(s):** Your Name  
**Date:** January 2025  
**Reviewers:** Engineering Team

[TOC]

---

## Summary

This RFC proposes replacing the current FIFO background job queue with a priority-based queue system to improve responsiveness for user-facing jobs while maintaining throughput for batch operations.

*Replace this with a 2–3 sentence description of your proposal.*

## Motivation

The current job queue processes all jobs in arrival order, regardless of urgency. This creates two problems:

1. **User-facing jobs are delayed** when the queue is backed up with batch work. Users wait 30+ seconds for email confirmations during high-traffic periods.
2. **Batch jobs are unpredictable** — they can't be scheduled to run during off-peak hours.

*Describe the specific pain point or opportunity. Include metrics where possible.*

## Detailed Design

### Priority Tiers

Jobs will be assigned to one of three priority tiers:

| Tier | Priority | Use Cases | Target Latency |
|------|----------|-----------|----------------|
| P0 | Critical | Auth emails, payment confirmations | < 5s |
| P1 | Normal | Notifications, exports | < 60s |
| P2 | Batch | Analytics aggregation, cleanup | Best effort |

### Implementation

Workers poll the queue starting at the highest priority tier. If no P0 jobs exist, they pull from P1, and so on.

```python
def poll_job():
    for priority in [Priority.P0, Priority.P1, Priority.P2]:
        job = queue.poll(priority=priority, timeout=1.0)
        if job:
            return job
    return None
```

### Migration Strategy

1. Deploy new priority queue system alongside the existing queue
2. Route new job submissions to the priority queue
3. Drain the old queue (allow existing jobs to complete)
4. Remove old queue infrastructure

Zero downtime. Estimated 2-week migration window.

## Drawbacks

- Adds operational complexity — two queues to monitor during the transition period
- P2 jobs could theoretically be starved if P0/P1 load is consistently high (mitigated by a minimum P2 throughput guarantee of 10 jobs/min)

*Describe the costs and risks of this approach.*

## Alternatives Considered

**Option A: Rate-limit batch jobs at ingestion**
Simpler than a full priority system, but doesn't help with burst scenarios where batch jobs already in the queue delay critical work.

**Option B: Separate worker pools per job type**
Provides strong isolation but doubles infrastructure cost and doesn't gracefully handle load shifts.

*Describe alternative approaches and why they were not chosen.*

## Unresolved Questions

- Should P0 jobs have a maximum execution time, after which they move to a dead-letter queue?
- Do we need a UI for operations to manually promote or demote jobs?

*List open questions that need answers before implementation begins.*

---

*Questions or feedback? Leave a comment or open a discussion thread.*