Backend2023

Distributed Certificate Generation Pipeline

Re-architected a large-scale document-generation workflow into a two-server processing pipeline, cutting runtime from ~2 days to ~6 hours.

Overview

A certificate-generation workflow for ~40,000 students previously ran on a single server and required roughly 2 days to complete. It was redesigned as a two-server processing pipeline with queue-based job tracking, bounded concurrency, retries, and locking, reducing runtime to ~6 hours.

Problem

The original single-server process handled document generation and conversion sequentially, creating a throughput bottleneck and making failure recovery difficult at volumes of tens of thousands of students.

Requirements

Reduce total processing time for ~40,000 students well below the existing ~2-day runtime
Process multiple documents concurrently without generating duplicate certificates
Retry transient conversion or network failures without restarting the entire batch
Separate document preparation from document conversion to remove the single-server bottleneck

Architecture

The workload was split across two servers. The first server loads student data, fills DOCX templates, uploads temporary documents to Microsoft 365 storage, and records processing status in MongoDB. A separate worker continuously processes pending jobs, converts DOCX files to PDF through Microsoft Graph with bounded concurrency, downloads the generated PDFs, uploads the final documents, and updates job status. Job locking prevents duplicate processing, while retry and backoff handling recover from transient failures.

Technical Challenges

Safely processing jobs concurrently

Parallel processing improved throughput but required locking and job-status transitions to ensure the same document was not processed by multiple workers.

Handling external document-conversion failures

Microsoft Graph conversion could fail because of throttling, timeouts, or unsupported documents, requiring different retry and failure-handling strategies.

Separating a tightly coupled single-server workflow

The original process combined data loading, document generation, conversion, and storage in one flow, so the redesign had to split responsibilities without losing job state or reliability.

Key Decisions

Split document preparation and conversion across two servers

Separating the CPU and I/O-heavy preparation stage from the external document-conversion stage removed the single-server bottleneck and allowed both stages to operate independently.

Use bounded concurrency for Microsoft Graph conversion

Conversion requests were limited to a controlled number of concurrent operations to increase throughput without overwhelming the external API.

Solution

Replaced the single-server workflow with a two-stage processing pipeline that independently prepares documents and converts them to PDF, with persistent job state, locking, bounded concurrency, retries, and failure classification.

Performance & Scalability

Reduced processing time for ~40,000 students from ~2 days to ~6 hours — an 87.5% reduction.

Results

Reduced end-to-end processing time from ~48 hours to ~6 hours.
Allowed individual failed jobs to retry without restarting the entire batch.
Prevented duplicate document processing through job locking.
Created an architecture that can be extended with additional workers if future load increases.

Lessons Learned

Large throughput improvements often come from redesigning the workflow rather than simply increasing server resources.
Once processing becomes concurrent, locking, retries, idempotency, and failure handling become critical parts of the architecture.