Unlimited OCR: One-Shot Long-Horizon Document Parsing with Constant Memory
“
The core question this article answers: Why do current OCR models slow down and run out of memory when processing long documents, and how does Unlimited OCR solve this by mimicking human working memory?
1. The Long-Horizon Parsing Problem
Core question: What makes long-document OCR so difficult for today’s models, and why does the human approach to copying books remain far superior?
Humans perform long-horizon parsing tasks with remarkable ease. We can transcribe hundreds of pages, translate hours of audio, or copy entire books without our cognitive efficiency degrading. We maintain a continuous state: distant outputs fade softly from memory while nearby context keeps us oriented. We do not re-read everything we have already written; we simply glance at the immediately surrounding text to stay on track.
Current OCR models cannot do this. Even state-of-the-art end-to-end systems process documents page-by-page in a for-loop, resetting memory at every step. This fragments a coherent long-horizon process into isolated short tasks managed by an external scheduler. It works, but it is an engineering workaround, not a step toward general intelligence.
The deeper issue lies in how large language model decoders handle attention. As output sequences lengthen, the KV cache grows linearly, driving up memory consumption and progressively slowing generation. Standard full attention requires every new token to attend to every previous token—a design that becomes prohibitively expensive when parsing dozens of pages in a single pass.
“
Author’s reflection: Reading this comparison, I was struck by how often we accept “page-by-page processing” as the natural state of OCR. We have normalized a workaround. The paper’s opening observation—that humans don’t reset their working memory every page—feels obvious in retrospect, yet it took this work to make it the central design principle of a new architecture.
2. Introducing Unlimited OCR and R-SWA
Core question: What is Unlimited OCR, and how does Reference Sliding Window Attention enable constant-memory decoding?
Unlimited OCR is an end-to-end document parsing model built on the DeepSeek OCR baseline. It replaces all attention layers in the decoder with Reference Sliding Window Attention (R-SWA), a mechanism designed to emulate human parsing working memory.
Figure 1: Reference Sliding Window Attention. Each generated token attends to all reference tokens (visual tokens and prompt) and only the preceding n output tokens (128 by default). Compared to standard full attention, R-SWA maintains constant KV cache. Compared to vanilla sliding window attention, it preserves visual token fidelity by excluding them from state transitions.
2.1 Architecture Overview
Unlimited OCR consists of two main components:
The DeepEncoder compresses a 1024×1024 PDF page to just 256 visual tokens. This high compression ratio is critical because visual tokens remain static throughout decoding—they are encoded once and never updated. The decoder then generates text using R-SWA, which keeps the KV cache bounded regardless of output length.
2.2 How R-SWA Works
R-SWA constrains attention to a two-segment window of size m + n:
-
Prefix window (m): Contains all visual tokens and the prompt. This window is globally visible to every generated token and remains fixed throughout inference. Its size depends only on the number of pages or document resolution. -
Causal sliding window (n): Contains the preceding n output tokens (default n = 128). This window slides forward as decoding progresses.
Formally, the accessible context for token t is:
\mathcal{N}(t) = \mathcal{P} \cup \mathcal{D}_n(t)$$
Where:
– $\mathcal{P} = \{1, \dots, L_m\}$ is the fixed prefix of length $L_m$
– $\mathcal{D}_n(t) = \{j \mid \max(L_m+1, L_m+t-n) \leq j \leq L_m+t-1\}$ is the causal sliding window of width n
The attention weights follow standard scaled dot-product attention, but only over this bounded set:
The output representation aggregates values over the same accessible set:
\mathbf{o}_t = \sum_{j \in \mathcal{N}(t)} \alpha_{tj} \mathbf{v}_j$$
> **Author’s reflection:** The elegance of this formulation is that it changes almost nothing about how attention computes—only about what it computes over. The model still uses familiar query-key-value operations. The innovation is architectural: deciding which keys and values deserve to stay in memory. This makes R-SWA both theoretically clean and practically deployable.
—
## 3. KV Cache: From Linear Growth to Constant Bound
**Core question:** How much memory does R-SWA actually save, and why does this matter for production deployment?
### 3.1 The Memory Problem in Standard Attention
Standard Multi-Head Attention (MHA) stores key-value pairs for every token generated. After generating T tokens, the KV cache size is:
This grows linearly with output length. For a 100K token output—which is realistic when parsing a 20-30 page document—this becomes a severe bottleneck.
3.2 R-SWA’s Constant Cache
Under R-SWA, the model always retains the full prefix cache of size , but for generated tokens it only keeps the most recent n:
C_{\text{R-SWA}}(T) = L_m + \min(n, T) \leq L_m + n$$
The cache ratio compared to standard MHA is:
When , this simplifies to:
\rho(T) \approx \frac{L_m + n}{T} \to 0
Practical implication: As output length grows, R-SWA’s memory footprint approaches a constant, while standard attention’s grows without bound.
3.3 Implementation: The Queue Design
The KV cache is implemented as a queue with capacity m + n. Each time a new token is generated, the KV corresponding to the (m+1)-th token in the queue is evicted. This FIFO eviction policy ensures both computational cost and memory usage remain stable throughout generation.
Figure 2: Unlimited OCR architecture. The model features a unified end-to-end design with an encoder and MoE-LLM decoder where all attention mechanisms are R-SWA. The KV cache operates as a queue with capacity m+n, evicting the oldest output token’s KV on each step.
“
Author’s reflection: The queue abstraction is worth appreciating. Many efficient attention variants require complex indexing or sparse data structures. A simple FIFO queue with fixed capacity is something every production engineer can reason about, debug, and optimize. The paper’s kernel study confirms this simplicity doesn’t come at a performance cost.
4. DeepEncoder: The High-Compression Visual Frontend
Core question: How does Unlimited OCR handle high-resolution document images without overwhelming the decoder?
4.1 Cascaded Attention Design
DeepEncoder, originally introduced in DeepSeek OCR, cascades two vision transformers:
-
SAM-ViT processes original image tokens using window attention only, keeping activation values low for high-resolution inputs -
CLIP-ViT operates on compressed tokens with global attention, capturing cross-image semantic relationships
At the bridge between them, a 16× token compression is applied. The first half relies entirely on window attention; global attention is reserved exclusively for compressed tokens.
4.2 Resolution Modes
Unlimited OCR retains two resolution configurations:
In Base mode, a 1024×1024 PDF image compresses to 256 tokens. For a 20-page document, this means approximately 5,120 visual tokens in the prefix—manageable within a 32K context window, leaving ample room for text generation.
4.3 Why Visual Tokens Must Stay Static
A critical design choice: visual tokens do not undergo state transitions alongside output tokens. They are encoded once and remain static throughout the entire long-horizon parsing process. This is why high compression is acceptable—the visual representation is not being repeatedly transformed and potentially degraded.
“
Author’s reflection: This “encode once, reference forever” principle distinguishes R-SWA from linear attention mechanisms. In linear attention variants, all tokens participate in recurrent state updates, which progressively blur visual features. The paper’s insight is that reference tokens and generated tokens deserve fundamentally different treatment. This seems obvious for OCR, but I wonder how many other multimodal tasks suffer from unnecessary state transitions on their input modalities.
5. Training and Data Pipeline
Core question: How was Unlimited OCR trained, and what resources were required?
5.1 Data Engine
The training corpus comprises approximately 2 million document OCR samples:
All data is packed into sequences of 32K tokens. The multi-page synthesis strategy is straightforward but effective: randomly concatenate single-page samples and insert page boundary markers.
5.2 Training Configuration
Starting from the DeepSeek OCR checkpoint, Unlimited OCR underwent continued training:
Key training decision: The DeepEncoder was frozen throughout training. Only LLM decoder parameters were updated. This is pragmatic—the encoder was already sufficiently optimized in DeepSeek OCR, and the bottleneck being addressed is purely in the decoding stage.
5.3 Inference Framework Support
R-SWA’s KV cache management has been implemented in:
-
Transformers library: Direct support for the queue-based cache eviction logic -
SGLang inference engine: Kernel-level optimizations for constant-throughput generation
Both frameworks enable Unlimited OCR to operate at constant tokens-per-second and constant GPU memory during inference.
“
Author’s reflection: The decision to freeze the encoder and train only 4,000 steps is telling. It suggests R-SWA is not just compatible with existing pretrained models—it actually unlocks capabilities that were latent in them. The baseline model already “knew” how to parse documents; it just couldn’t do so efficiently at length. This makes R-SWA an attractive upgrade path for production systems already running DeepSeek OCR or similar architectures.
6. Performance Evaluation
Core question: Does R-SWA maintain accuracy while delivering efficiency gains, or is there a trade-off?
6.1 OmniDocBench v1.5: Main Results
OmniDocBench evaluates document parsing across multiple dimensions: text recognition, formula recognition, table structure extraction, and reading order prediction.
Improvement over baseline: +6.22 overall, -0.035 text edit distance, +9.24 formula CDM, +5.96 table TEDS, -0.041 read order edit distance.
6.2 OmniDocBench v1.6: Current SOTA Comparison
Unlimited OCR achieves end-to-end SOTA on v1.6 with only 3B total parameters and 0.5B activated parameters—significantly smaller than many competitors.
6.3 Subcategory Analysis
Across nine document types in OmniDocBench v1.5, Unlimited OCR shows consistent gains:
Text Edit Distance (lower is better):
Reading Order Edit Distance (lower is better):
“
Author’s reflection: The newspaper and note categories show particularly dramatic improvements. I suspect this is because these document types have highly irregular layouts where global attention can actually mislead the model. R-SWA’s local focus forces the model to attend to spatially nearby content, which aligns better with how humans read such documents. The “free lunch” observation in the paper—that R-SWA improves accuracy while reducing cost—is genuinely surprising and deserves more theoretical investigation.
7. Long-Horizon Parsing in Practice
Core question: Can Unlimited OCR really parse dozens of pages in one shot, and does quality hold up?
7.1 Multi-Page Test Results
An in-house benchmark was constructed using novels, documents, and papers divided by page count:
Key observations:
-
Distinct-n scores remain above 96% even at 40+ pages, indicating minimal repetitive generation -
Edit distance stays below 0.11 across all tested lengths -
Errors at 40+ pages primarily stem from small text in PDFs being difficult to discern at 1024×1024 resolution—not from R-SWA losing track of parsing progress
Application scenario: A legal tech company needs to digitize case files averaging 30-50 pages. Traditional OCR requires breaking files into chunks, processing separately, and reassembling—with potential context loss at chunk boundaries. Unlimited OCR can ingest the entire file in one forward pass, maintaining reading order and cross-page references without manual intervention.
7.2 Inference Speed Stability
Theoretical TPS (tokens per second) comparison under ideal concurrency:
At short outputs, speeds are comparable. As length grows, DeepSeek OCR’s TPS declines steadily while Unlimited OCR’s remains stable. The crossover point where R-SWA pays for itself appears to be around 512 tokens.
Figure 3: Flash Attention v3 kernel latency as decoding length increases. Standard MHA latency grows with each step, while R-SWA latency remains constant. The spike in DeepSeek OCR occurs when KV cache length crosses an alignment boundary, causing data transfer efficiency to drop—an issue that does not arise with R-SWA.
“
Author’s reflection: The 35% speed advantage at 6K tokens is substantial, but I find the stability even more valuable for production. Predictable latency means predictable SLAs. When you promise a customer that 50 pages will process in under 30 seconds, you need that time to be reliable regardless of whether the document is 10 pages or 50. R-SWA’s constant latency makes this possible.
8. Efficiency Analysis and Production Implications
Core question: What does constant KV cache mean for real-world deployment costs?
8.1 Memory Footprint
For a 20-page document with 5,120 visual tokens and ~50,000 output tokens:
8.2 Throughput Under Concurrency
In production OmniDocBench testing:
-
Unlimited OCR: 5,580 TPS at 512 concurrency -
DeepSeek OCR: 4,951 TPS under identical conditions -
Improvement: +12.7%
The paper notes that OmniDocBench documents are relatively short on average. The longer the typical output, the more pronounced Unlimited OCR’s advantage becomes.
Application scenario: A cloud OCR service processes millions of pages daily. With standard attention, peak loads on long documents require aggressive auto-scaling because latency degrades unpredictably. With R-SWA, throughput remains constant, allowing tighter resource provisioning and more predictable pricing for customers.
9. Limitations and Future Directions
Core question: What can’t Unlimited OCR do yet, and where is this technology heading?
9.1 Current Limitations
-
Prefill length constraints: While output length is effectively unlimited, the input (prefill) is still bounded by the model’s maximum context length. At 32K, this limits how many pages can be encoded in one pass.
-
Resolution trade-offs: Multi-page mode uses 1024×1024 resolution. Small text may be difficult to discern compared to single-page high-resolution modes.
9.2 Short-Term Roadmap
-
Train models with 128K context length to support more pages in the prefill stage
9.3 Long-Term Vision
-
Prefill pool architecture: Enable the model to automatically fetch prefill KV chunks, simulating human page-flipping behavior -
Cross-task transfer: Apply R-SWA to ASR, translation, and other reference-based long-horizon tasks -
True unlimited parsing: Break the finite context barrier through intelligent KV chunk retrieval
“
Author’s reflection: The prefill pool concept is particularly intriguing. It suggests a future where the model doesn’t just process long documents—it navigates them. Instead of loading 100 pages into context, the model might hold 10 pages actively and “reach for” others as needed. This would be closer to human reading behavior than even the current one-shot approach.
10. Conclusion
Core question: What is the fundamental insight of Unlimited OCR, and why does it matter beyond OCR?
Unlimited OCR demonstrates that replacing all standard attention in an end-to-end model’s decoder with causal reference-based sliding window attention yields not just efficiency gains, but accuracy improvements on parsing tasks. The model learns to pass useful information continuously through the sliding window, and this soft form of forgetting aligns with human cognitive behavior during long-horizon tasks.
The key contributions are:
-
R-SWA mechanism: Maintains constant KV cache during decoding by attending globally to reference tokens and locally to recent outputs -
Unlimited OCR model: Achieves one-shot parsing of dozens of pages with 93%+ accuracy on OmniDocBench, outperforming the DeepSeek OCR baseline by 6% -
General applicability: R-SWA is a general-purpose parsing attention mechanism extensible to ASR, translation, and other reference-based tasks
The broader significance is architectural: rather than brute-force scaling context length, Unlimited OCR identifies an elegant attention pattern that achieves long-horizon capability through bounded memory and computation. This represents a different path toward efficient long-context AI—one that mimics human cognitive constraints rather than attempting to transcend them.
Action Checklist / Implementation Steps
For engineers considering R-SWA adoption:
-
Assess your task: Confirm it follows the reference-generation pattern (static input, dynamic long output) -
Baseline selection: If using DeepSeek OCR or similar end-to-end VLM, R-SWA can replace decoder attention directly -
Window sizing: Start with n=128; increase for tasks requiring longer local dependencies -
Cache implementation: Use a FIFO queue of capacity m+n; evict position (m+1) on each new token -
Framework integration: Transformers and SGLang already support R-SWA; custom kernels can follow the same queue logic -
Resolution planning: Use Base mode (1024×1024) for multi-page, Gundam mode for single-page precision -
Training strategy: Freeze encoder, continue-train decoder for ~4K steps from a strong OCR checkpoint -
Production monitoring: Track TPS stability across output lengths; expect constant performance beyond 512 tokens
One-Page Overview
Frequently Asked Questions
Q1: Does R-SWA’s 128-token window cause the model to “forget” what it’s doing?
No. Empirical results on nine document types show Unlimited OCR outperforms full-attention baselines. The model learns to maintain parsing state through the sliding window’s continuous information flow. The bounded window actually improves focus by preventing attention dilution.
Q2: How many pages can Unlimited OCR process in one pass?
At 32K context, tested up to 40+ pages with strong results. The practical limit is prefill length (visual tokens), not output length. A 128K context version is planned to extend this further.
Q3: What’s the difference between R-SWA and linear attention (Mamba, RWKV)?
Linear attention applies recurrent state updates to all tokens, progressively blurring visual features. R-SWA excludes reference tokens from state transitions—they remain static—while using a causal window only for generated tokens. This preserves visual fidelity while achieving constant memory.
Q4: How much training is needed to adopt R-SWA?
From a DeepSeek OCR checkpoint: 4,000 steps on 8×16 A800 GPUs, freezing the encoder and training only decoder parameters. This is efficient enough for most research labs and many production teams.
Q5: Is R-SWA only useful for OCR?
No. It is a general parsing attention mechanism applicable to any task with static reference input and dynamic long output, including speech recognition (ASR), machine translation, video captioning, and long-document summarization.
Q6: Why does limiting attention improve accuracy instead of hurting it?
Full attention can diverge as output length increases, with the model attending to irrelevant distant tokens. R-SWA’s local focus forces more efficient information routing through the window. Historical information is causally and continuously fed forward, enabling clear progress tracking.
Q7: What causes errors in 40+ page parsing?
Most errors occur with small text that is difficult to discern at 1024×1024 resolution in Base mode. These are encoder resolution limitations, not R-SWA losing direction. The model maintains coherent parsing state throughout.
Q8: Where can I access the model and code?
Code and model weights are publicly available at the project’s GitHub repository. Both Transformers and SGLang inference frameworks are supported with constant-throughput generation.
