Backend2023

Certificate Generation Pipeline Re-Architecture

Re-architected a single-server certificate-generation job into a queue-based, two-server pipeline — cutting processing from ~2 days to ~6 hours.

Overview

A certificate-generation job for ~40,000 students ran on a single server and took roughly 2 days to complete. It was re-architected into a queue-based, two-server pipeline with concurrency control, retries, and locking, cutting runtime to ~6 hours.

Problem

The existing single-server certificate-generation process couldn't keep up with a volume of ~40,000 students, taking around 2 days per run and leaving little room for retries or failure recovery.

Requirements

Reduce total processing time for ~40,000 students well below the existing ~2-day runtime
Guarantee each student's certificate is generated exactly once, even under concurrent processing
Handle transient failures with retries instead of failing an entire run

Architecture

A queue-based pipeline split across two servers, with concurrency control to safely parallelize work, distributed locking to prevent duplicate processing of the same student, and retry handling for transient failures.

Technical Challenges

Safe concurrency across two servers

Splitting work across two servers meant introducing locking to guarantee no student's certificate was generated twice while still processing in parallel.

Recovering from failures without restarting the whole run

A queue-based design let individual failed jobs retry independently instead of forcing a full re-run across all ~40,000 students.

Key Decisions

Queue-based two-server pipeline instead of a faster single server

Moving to a distributed, queue-based design added complexity (locking, retries) but was necessary to get meaningfully past the throughput ceiling of a single server.

Solution

Replaced the single-server job with a queue-based, two-server pipeline that adds concurrency control, retries, and locking, so certificates generate safely in parallel instead of serially.

Performance & Scalability

Cut total processing time for ~40,000 students from ~2 days to ~6 hours (87.5% faster).

Results

87.5% reduction in processing time (~2 days → ~6 hours) for ~40,000 students.

Lessons Learned

TODO: Add a personal lesson learned from this re-architecture.