How to generate 10,000 PDFs without running out of memory

The first bulk PDF job every team writes looks the same: a loop, a browser launch inside it, and a list collecting the results. It works for a hundred documents and dies at two thousand. The fixes are not clever, but they are specific, and they come in a particular order.

Failure one: a browser per document

Launching Chromium costs one to two seconds and a few hundred megabytes. Do that ten thousand times and you have spent three hours starting browsers before rendering anything.

Keep one browser per worker process and open a fresh context per document. A context is cheap — single-digit milliseconds — and gives you the isolation people think they need a new browser for: separate cookies, storage and cache.

# once per process
browser = playwright.chromium.launch(args=["--no-sandbox", "--disable-dev-shm-usage"])

# per document
context = browser.new_context()
page = context.new_page()
page.set_content(html, wait_until="networkidle")
pdf = page.pdf(format="A4")
context.close()

Measured on our fleet: cold start 1.94 s, warm render 0.64 s. That ratio is the whole argument.

Failure two: unbounded concurrency

Promise.all or asyncio.gather over ten thousand renders asks the machine for ten thousand pages at once. Chromium does not refuse — it tries, swaps, and the OOM killer picks a victim, usually your web server rather than the browser.

Use a queue with a fixed number of consumers. Four to six concurrent renders per 8 GB is a sensible starting point; measure and adjust. Our own numbers: two worker processes with concurrency three each sustain 8 requests/second at p95 0.92 s on an 8-core box.

Failure three: holding results in memory

A 300 KB PDF times ten thousand is 3 GB sitting in a list. Write each result to disk or object storage the moment it exists, keep only the path, and stream a ZIP at the end if the user wants one.

path = storage.save(doc_id, pdf)     # bytes leave the process here
results.append(path)                 # not the bytes
del pdf

Failure four: the leak you did not cause

Long-lived Chromium processes grow. Not fast, but a browser that has rendered five thousand pages holds noticeably more than one that has rendered five — fragmentation, cached fonts, GPU buffers.

Recycle the browser every few hundred renders. It costs one cold start per cycle and buys a flat memory graph:

RENDERS_PER_BROWSER = 200

if renders_since_launch >= RENDERS_PER_BROWSER:
    browser.close()
    browser = launch()
    renders_since_launch = 0

Celery users get this free with --max-tasks-per-child.

Failure five: one bad document poisons the batch

A page that never fires load, a font server that hangs, an infinite JavaScript loop. Every render needs a timeout, and a timed-out render must fail alone — logged, retried once on a fresh browser, then marked failed — without blocking the queue.

try:
    pdf = render(html, timeout=30)
except RenderTimeout:
    restart_browser()          # it may be wedged
    if attempt < 1:
        raise Retry(countdown=2)
    mark_failed(doc_id, "timed out after 30 s")

The restart_browser() line matters. A browser that timed out once often times out again; a fresh one usually succeeds.

The pipeline shape

Putting the five together:

1. The request enqueues one job per document and returns immediately with a batch id 2. A fixed pool of render workers, each owning one browser, each recycling after N documents 3. Every result written to storage with a TTL; the job row keeps the path 4. A webhook or a status endpoint tells the caller when the batch is done 5. Failures are per-document, visible, and retryable

Sizing it

Rough arithmetic for planning, based on 0.65 s per simple document:

Complex documents with charts and web fonts run 2–4× slower, so scale accordingly.

If you would rather not build it

This is the pipeline PDFGeny runs, and one request uses it:

curl -X POST https://pdfgeny.com/api/v1/batch \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"documents":[
        {"template":"certificate","data":{"recipient":"Ada Lovelace"}},
        {"template":"certificate","data":{"recipient":"Grace Hopper"}}
      ]}'

Up to 100 documents per call, a signed download link per document valid for seven days, and a webhook when each one finishes. A worked example for a full cohort is in issuing 5,000 course certificates.