Editorial illustration for Microservice Architecture Unites OCR, Classification, and LLM Pipelines
Microservice Architecture Unites OCR, Classification,...
Every AI demo pretends the work ends when the model works. The real work begins when you try to run it. You find the bottleneck is never the flashy part.
Take document understanding. You build a pipeline to classify, read, and extract data from thousands of forms. On paper, the large language model is the star.
In production, something else becomes the villain. We built one of these systems and found the problem. It’s almost never the model you think.
To close that gap, we present a microservice architecture that encapsulates pipelines of multiple models for classification, optical character recognition (OCR), and large language model structured field extraction as well as our experience running this pipeline on thousands of multi-page documents per hour. We describe our primary design decisions, including a hybrid classification, separation of GPU-bound inference from CPU-bound orchestration, use of asynchronous processing for the many IO-bound operations in the pipeline, and an independent, horizontal scaling strategy. Using batch profiling, we identified two surprising qualitative findings that shape production deployments: OCR, not language-model parsing, dominates end-to-end latency, and the system saturates at a concurrency determined by shared GPU-inference capacity rather than worker count. Our goal is to provide practitioners with concrete architectural patterns for building document understanding systems that work beyond the benchmark; effectively operationalizing models in production.
The key was treating each piece of the pipeline as its own scaling problem. We split GPU tasks from CPU tasks. We made everything that could wait asynchronous. This let the system scale in the weird, lopsided way real workloads demand.
The profiling data told a clear story. OCR was the slowest step by a wide margin, consuming more time than all the LLM parsing combined. And the whole thing hit a wall not when we ran out of servers, but when we ran out of shared GPU power for those early stages. Your ceiling is set by your heaviest shared resource.
This flips the script. Building for scale means ignoring the benchmark leaderboard and instrumenting your own pipeline. You design around the slowest, most expensive step.
You stop asking which model is smartest and start asking which one makes the whole line stop. That is the only metric that matters outside the lab.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives Proposal Calls for Data Probes to Study Impact of Training Data on LLMs
From Archives Isotonic calibration gets O(n⁻¹/³) sample complexity, cost‑optimal LLM routing
Common Questions Answered
Why is OCR the bottleneck in document understanding pipelines rather than the LLM?
According to the profiling data from their production system, OCR was the slowest step by a wide margin, consuming more time than all the LLM parsing combined. This demonstrates that the flashy AI component (the language model) is rarely the actual performance bottleneck in real-world document processing pipelines.
How does microservice architecture help scale document classification and extraction workflows?
By treating each piece of the pipeline as its own scaling problem and splitting GPU tasks from CPU tasks, the microservice approach allows systems to scale in the lopsided way real workloads demand. This separation enables asynchronous processing for components that can wait, preventing one slow service from blocking the entire pipeline.
What was the actual scaling limitation discovered in the document understanding system?
The system hit a wall not when it ran out of servers, but when it ran out of shared GPU power for the early pipeline stages like OCR. This revealed that infrastructure constraints, rather than algorithmic limitations, became the real bottleneck in production deployment.
Why is making pipeline components asynchronous important for document processing systems?
Asynchronous processing allows components that don't need immediate results to operate independently, preventing faster stages from waiting on slower ones like OCR. This architectural pattern is essential for handling the uneven performance characteristics of real document understanding workloads.
Further Reading
- A Microservice Architecture for OCR and LLM Pipelines in Production — arXiv
- Beyond OCR: How AI is Transforming Document Processing and Understanding — InfoQ
- Building a Scalable OCR Pipeline: A Technical Guide — HealthEdge
- Hybrid OCR-LLM: Not a Bigger Model, but a Smarter Pipeline — AI Exp Journey
- Monolith vs micro: The $1M ML design decision — DecodingAI
