Building a Job Queue System with BullMQ and Redis

Why You Need a Job Queue

Not everything can happen in the request-response cycle. Sending emails, processing images, generating reports, syncing data, running AI inference — these tasks take seconds or minutes, and your users should not wait.

<div style="margin:2.5rem auto;max-width:600px;width:100%;text-align:center;"><svg viewBox="0 0 600 180" xmlns="http://www.w3.org/2000/svg" style="width:100%;height:auto;"><rect width="600" height="180" rx="12" fill="#1a1a2e"/><rect x="30" y="60" width="80" height="50" rx="25" fill="#3b82f6" opacity="0.85"/><text x="70" y="90" text-anchor="middle" fill="#ffffff" font-size="11" font-family="system-ui">Prompt</text><rect x="145" y="50" width="90" height="70" rx="8" fill="#6366f1" opacity="0.85"/><text x="190" y="80" text-anchor="middle" fill="#ffffff" font-size="10" font-family="system-ui">Embed</text><text x="190" y="95" text-anchor="middle" fill="#ffffff" font-size="10" font-family="system-ui">[0.2, 0.8...]</text><rect x="270" y="50" width="90" height="70" rx="8" fill="#a855f7" opacity="0.85"/><text x="315" y="75" text-anchor="middle" fill="#ffffff" font-size="10" font-family="system-ui">Vector</text><text x="315" y="90" text-anchor="middle" fill="#ffffff" font-size="10" font-family="system-ui">Search</text><text x="315" y="105" text-anchor="middle" fill="#ffffff" font-size="9" font-family="system-ui" opacity="0.7">top-k=5</text><rect x="395" y="50" width="90" height="70" rx="8" fill="#2dd4bf" opacity="0.85"/><text x="440" y="80" text-anchor="middle" fill="#1a1a2e" font-size="11" font-family="system-ui" font-weight="bold">LLM</text><text x="440" y="95" text-anchor="middle" fill="#1a1a2e" font-size="9" font-family="system-ui">+ context</text><rect x="520" y="60" width="55" height="50" rx="25" fill="#f59e0b" opacity="0.85"/><text x="547" y="90" text-anchor="middle" fill="#1a1a2e" font-size="10" font-family="system-ui">Reply</text><defs><marker id="arrow4" markerWidth="8" markerHeight="6" refX="8" refY="3" orient="auto"><path d="M0,0 L8,3 L0,6" fill="#e2e8f0"/></marker></defs><line x1="112" y1="85" x2="143" y2="85" stroke="#e2e8f0" stroke-width="1.5" marker-end="url(#arrow4)"/><line x1="237" y1="85" x2="268" y2="85" stroke="#e2e8f0" stroke-width="1.5" marker-end="url(#arrow4)"/><line x1="362" y1="85" x2="393" y2="85" stroke="#e2e8f0" stroke-width="1.5" marker-end="url(#arrow4)"/><line x1="487" y1="85" x2="518" y2="85" stroke="#e2e8f0" stroke-width="1.5" marker-end="url(#arrow4)"/><text x="300" y="155" text-anchor="middle" fill="#94a3b8" font-size="10" font-family="system-ui">Retrieval-Augmented Generation (RAG) Flow</text></svg><p style="margin-top:0.75rem;font-size:0.85rem;color:#94a3b8;font-style:italic;line-height:1.4;">RAG architecture: user prompts are embedded, matched against a vector store, then fed to an LLM with retrieved context.</p></div>

A job queue lets you push work into a background process that runs independently. BullMQ is the most popular Node.js/TypeScript job queue, built on Redis. It handles millions of jobs per day at companies like Mozilla, Automattic, and many others.

Setup

npm install bullmq ioredis

// src/queue/connection.ts
import { Redis } from 'ioredis';

export const connection = new Redis({
  host: process.env.REDIS_HOST || 'localhost',
  port: parseInt(process.env.REDIS_PORT || '6379'),
  maxRetriesPerRequest: null,  // Required by BullMQ
});

Creating Queues and Producers

// src/queue/email.queue.ts
import { Queue } from 'bullmq';
import { connection } from './connection';

interface EmailJobData {
  to
  subject
  template
  variables: Record<string, string>;
}

export const emailQueue = new Queue<EmailJobData>('email', {
  connection,
  defaultJobOptions: {
    attempts: 3,
    backoff: {
      type: 'exponential',
      delay: 2000,  // 2s, 4s, 8s
    },
    removeOnComplete: { count: 1000 },  // Keep last 1000 completed
    removeOnFail: { count: 5000 },      // Keep last 5000 failed
  },
});

// Add a job
await emailQueue.add('welcome', {
  to: '[email protected]',
  subject: 'Welcome to TechSaaS',
  template: 'welcome',
  variables: { name: 'Alice' },
});

// Add a delayed job (send in 1 hour)
await emailQueue.add('reminder', {
  to: '[email protected]',
  subject: 'Complete your setup',
  template: 'reminder',
  variables: { name: 'Alice' },
}, {
  delay: 60 * 60 * 1000,  // 1 hour in ms
});

// Add a scheduled recurring job
await emailQueue.upsertJobScheduler(
  'weekly-digest',
  { pattern: '0 9 * * MON' },  // Every Monday at 9am
  {
    name: 'digest',
    data: { template: 'weekly-digest' },
  }
);

Creating Workers (Consumers)

// src/queue/email.worker.ts
import { Worker, Job } from 'bullmq';
import { connection } from './connection';
import { sendEmail } from '../services/email';
import { renderTemplate } from '../services/templates';

const emailWorker = new Worker<EmailJobData>(
  'email',
  async (job: Job<EmailJobData>) => {
    const { to, subject, template, variables } = job.data;

    // Update progress
    await job.updateProgress(10);

    // Render template
    const html = await renderTemplate(template, variables);
    await job.updateProgress(50);

    // Send email
    const result = await sendEmail({ to, subject, html });
    await job.updateProgress(100);

    // Return result (stored with completed job)
    return { messageId: result.messageId, sentAt: new Date().toISOString() };
  },
  {
    connection,
    concurrency: 5,        // Process 5 emails simultaneously
    limiter: {
      max: 50,             // Max 50 jobs
      duration: 60000,     // Per minute (respect email provider limits)
    },
  }
);

// Event handlers
emailWorker.on('completed', (job, result) => {
  console.log(`Email sent to ${job.data.to}: ${result.messageId}`);
});

emailWorker.on('failed', (job, error) => {
  console.error(`Email to ${job?.data.to} failed: ${error.message}`);
  // Alert if all retries exhausted
  if (job && job.attemptsMade >= (job.opts.attempts || 3)) {
    alertOpsTeam(`Email permanently failed: ${job.data.to}`);
  }
});

emailWorker.on('error', (error) => {
  console.error('Worker error:', error);
});

Job Priorities

// Priority: lower number = higher priority
await emailQueue.add('password-reset', data, { priority: 1 });    // Critical
await emailQueue.add('order-confirmation', data, { priority: 5 }); // Normal
await emailQueue.add('newsletter', data, { priority: 10 });        // Low

<div style="margin:2.5rem auto;max-width:600px;width:100%;text-align:center;"><svg viewBox="0 0 600 180" xmlns="http://www.w3.org/2000/svg" style="width:100%;height:auto;"><rect width="600" height="180" rx="12" fill="#1a1a2e"/><ellipse cx="150" cy="55" rx="60" ry="18" fill="#6366f1" opacity="0.8"/><rect x="90" y="55" width="120" height="50" fill="#6366f1" opacity="0.8"/><ellipse cx="150" cy="105" rx="60" ry="18" fill="#6366f1" opacity="0.9"/><text x="150" y="85" text-anchor="middle" fill="#ffffff" font-size="12" font-family="system-ui" font-weight="bold">Primary</text><text x="150" y="140" text-anchor="middle" fill="#94a3b8" font-size="10" font-family="system-ui">Read + Write</text><ellipse cx="400" cy="30" rx="50" ry="14" fill="#a855f7" opacity="0.7"/><rect x="350" y="30" width="100" height="35" fill="#a855f7" opacity="0.7"/><ellipse cx="400" cy="65" rx="50" ry="14" fill="#a855f7" opacity="0.8"/><text x="400" y="52" text-anchor="middle" fill="#ffffff" font-size="10" font-family="system-ui">Replica 1</text><ellipse cx="400" cy="110" rx="50" ry="14" fill="#a855f7" opacity="0.7"/><rect x="350" y="110" width="100" height="35" fill="#a855f7" opacity="0.7"/><ellipse cx="400" cy="145" rx="50" ry="14" fill="#a855f7" opacity="0.8"/><text x="400" y="132" text-anchor="middle" fill="#ffffff" font-size="10" font-family="system-ui">Replica 2</text><defs><marker id="arrow8" markerWidth="8" markerHeight="6" refX="8" refY="3" orient="auto"><path d="M0,0 L8,3 L0,6" fill="#2dd4bf"/></marker></defs><path d="M212,65 Q280,30 348,48" stroke="#2dd4bf" stroke-width="1.5" fill="none" marker-end="url(#arrow8)"/><path d="M212,90 Q280,130 348,128" stroke="#2dd4bf" stroke-width="1.5" fill="none" marker-end="url(#arrow8)"/><text x="280" y="55" text-anchor="middle" fill="#2dd4bf" font-size="9" font-family="system-ui">WAL stream</text><text x="280" y="130" text-anchor="middle" fill="#2dd4bf" font-size="9" font-family="system-ui">WAL stream</text><text x="500" y="52" text-anchor="start" fill="#94a3b8" font-size="9" font-family="system-ui">Read-only</text><text x="500" y="132" text-anchor="start" fill="#94a3b8" font-size="9" font-family="system-ui">Read-only</text></svg><p style="margin-top:0.75rem;font-size:0.85rem;color:#94a3b8;font-style:italic;line-height:1.4;">Database replication: the primary handles writes while replicas serve read queries via WAL streaming.</p></div>

Flow (Job Dependencies)

BullMQ supports parent-child job flows:

import { FlowProducer } from 'bullmq';

const flowProducer = new FlowProducer({ connection });

// Child jobs run first, parent runs after all children complete
await flowProducer.add({
  name: 'generate-report',
  queueName: 'reports',
  data: { reportId: 'RPT-001' },
  children: [
    {
      name: 'fetch-sales-data',
      queueName: 'data-fetch',
      data: { source: 'sales', dateRange: '2025-11' },
    },
    {
      name: 'fetch-user-data',
      queueName: 'data-fetch',
      data: { source: 'users', dateRange: '2025-11' },
    },
    {
      name: 'fetch-analytics',
      queueName: 'data-fetch',
      data: { source: 'analytics', dateRange: '2025-11' },
    },
  ],
});
// fetch-sales-data, fetch-user-data, fetch-analytics run in parallel
// generate-report runs after all three complete

Multiple Queue Pattern

Organize work into separate queues by type:

// src/queue/index.ts
import { Queue } from 'bullmq';
import { connection } from './connection';

export const queues = {
  email: new Queue('email', { connection }),
  imageProcessing: new Queue('image-processing', { connection }),
  dataSync: new Queue('data-sync', { connection }),
  aiInference: new Queue('ai-inference', { connection }),
  webhookDelivery: new Queue('webhook-delivery', { connection }),
};

// Different concurrency per queue type
// Email: high concurrency, I/O bound
// Image processing: low concurrency, CPU bound
// AI inference: single concurrency, GPU bound

Error Handling and Dead Letter Queue

const worker = new Worker('critical-tasks', async (job) => {
  try {
    await processJob(job.data);
  } catch (error) {
    if (error instanceof TransientError) {
      // Retry-able error: throw to trigger BullMQ retry
      throw error;
    }

    if (error instanceof PermanentError) {
      // Non-retryable: move to dead letter queue
      await deadLetterQueue.add('failed-job', {
        originalQueue: 'critical-tasks',
        originalJobId: job.id,
        data: job.data,
        error: error.message,
        failedAt: new Date().toISOString(),
      });

      // Don't throw — mark as completed to prevent retries
      return { status: 'moved_to_dlq', error: error.message };
    }

    throw error;  // Unknown error: let BullMQ retry
  }
}, { connection, concurrency: 3 });

Monitoring with Bull Board

// src/monitoring/bull-board.ts
import { createBullBoard } from '@bull-board/api';
import { BullMQAdapter } from '@bull-board/api/bullMQAdapter';
import { ExpressAdapter } from '@bull-board/express';

const serverAdapter = new ExpressAdapter();
serverAdapter.setBasePath('/admin/queues');

createBullBoard({
  queues: [
    new BullMQAdapter(emailQueue),
    new BullMQAdapter(imageQueue),
    new BullMQAdapter(aiQueue),
  ],
  serverAdapter,
});

// Mount on your Express app
app.use('/admin/queues', serverAdapter.getRouter());

<div style="margin:2.5rem auto;max-width:600px;width:100%;text-align:center;"><svg viewBox="0 0 600 200" xmlns="http://www.w3.org/2000/svg" style="width:100%;height:auto;"><rect width="600" height="200" rx="12" fill="#1a1a2e"/><text x="80" y="25" text-anchor="middle" fill="#94a3b8" font-size="10" font-family="system-ui">Input</text><circle cx="80" cy="50" r="14" fill="none" stroke="#3b82f6" stroke-width="2"/><circle cx="80" cy="100" r="14" fill="none" stroke="#3b82f6" stroke-width="2"/><circle cx="80" cy="150" r="14" fill="none" stroke="#3b82f6" stroke-width="2"/><text x="230" y="25" text-anchor="middle" fill="#94a3b8" font-size="10" font-family="system-ui">Hidden</text><circle cx="230" cy="45" r="14" fill="#6366f1" opacity="0.8"/><circle cx="230" cy="85" r="14" fill="#6366f1" opacity="0.8"/><circle cx="230" cy="125" r="14" fill="#6366f1" opacity="0.8"/><circle cx="230" cy="165" r="14" fill="#6366f1" opacity="0.8"/><text x="380" y="25" text-anchor="middle" fill="#94a3b8" font-size="10" font-family="system-ui">Hidden</text><circle cx="380" cy="55" r="14" fill="#a855f7" opacity="0.8"/><circle cx="380" cy="100" r="14" fill="#a855f7" opacity="0.8"/><circle cx="380" cy="145" r="14" fill="#a855f7" opacity="0.8"/><text x="520" y="25" text-anchor="middle" fill="#94a3b8" font-size="10" font-family="system-ui">Output</text><circle cx="520" cy="80" r="14" fill="none" stroke="#2dd4bf" stroke-width="2"/><circle cx="520" cy="130" r="14" fill="none" stroke="#2dd4bf" stroke-width="2"/><line x1="94" y1="50" x2="216" y2="45" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="94" y1="50" x2="216" y2="85" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="94" y1="50" x2="216" y2="125" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="94" y1="50" x2="216" y2="165" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="94" y1="100" x2="216" y2="45" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="94" y1="100" x2="216" y2="85" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="94" y1="100" x2="216" y2="125" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="94" y1="100" x2="216" y2="165" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="94" y1="150" x2="216" y2="45" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="94" y1="150" x2="216" y2="85" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="94" y1="150" x2="216" y2="125" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="94" y1="150" x2="216" y2="165" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="244" y1="45" x2="366" y2="55" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="244" y1="45" x2="366" y2="100" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="244" y1="45" x2="366" y2="145" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="244" y1="85" x2="366" y2="55" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="244" y1="85" x2="366" y2="100" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="244" y1="85" x2="366" y2="145" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="244" y1="125" x2="366" y2="55" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="244" y1="125" x2="366" y2="100" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="244" y1="125" x2="366" y2="145" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="244" y1="165" x2="366" y2="55" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="244" y1="165" x2="366" y2="100" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="244" y1="165" x2="366" y2="145" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="394" y1="55" x2="506" y2="80" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="394" y1="55" x2="506" y2="130" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="394" y1="100" x2="506" y2="80" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="394" y1="100" x2="506" y2="130" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="394" y1="145" x2="506" y2="80" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/><line x1="394" y1="145" x2="506" y2="130" stroke="#e2e8f0" stroke-width="0.5" opacity="0.3"/></svg><p style="margin-top:0.75rem;font-size:0.85rem;color:#94a3b8;font-style:italic;line-height:1.4;">Neural network architecture: data flows through input, hidden, and output layers.</p></div>

Production Checklist

1. Set `maxRetriesPerRequest: null` on Redis connection (BullMQ requirement) 2. Always set `removeOnComplete` and `removeOnFail` to prevent Redis memory exhaustion 3. Use exponential backoff for retries 4. Monitor queue depth — growing queues mean workers cannot keep up 5. Graceful shutdown: Call worker.close() on SIGTERM 6. Separate worker processes from API servers for reliability 7. Set appropriate concurrency per queue type

// Graceful shutdown
process.on('SIGTERM', async () => {
  console.log('Shutting down workers...');
  await emailWorker.close();
  await imageWorker.close();
  process.exit(0);
});

BullMQ with Redis is the backbone of background processing in the Node.js ecosystem. At TechSaaS, we use it for everything from email delivery to AI inference scheduling.