Unlimited OCR: Solving Memory Bottlenecks in Long-Horizon Parsing
Humans have a unique ability to process long documents. We do not need to restart our cognitive state every time we turn a page. We maintain a continuous understanding of the content while keeping the original source in view. Current end-to-end OCR models generally fail to replicate this. They struggle with long sequences, often relying on inefficient page-by-page processing that fragments coherent tasks and leads to escalating memory consumption.
Unlimited OCR changes this paradigm by introducing a new mechanism that allows models to parse dozens of pages in a single forward pass, all while keeping computational costs stable.
The Problem with KV Cache Growth
The main obstacle to processing long documents in modern transformers is the KV cache. As a model generates text, it stores key and value vectors for every preceding token. Over time, this cache grows linearly with the length of the output. This results in two major issues: memory consumption increases rapidly, and generation speed slows down significantly as the sequence length grows.
Most current systems work around this by using a for-loop approach, processing one page at a time. This resets the memory for each page and destroys the continuity of the task. We needed a way to maintain context without the runaway cost of a standard attention mechanism.
Reference Sliding Window Attention (R-SWA)
Unlimited OCR addresses this by replacing standard attention layers with Reference Sliding Window Attention (R-SWA). This design draws inspiration from human working memory.
R-SWA applies two distinct rules to manage what the model attends to:
- The model maintains access to all reference tokens, which includes the original visual tokens from the image and the prompt.
- The model limits its attention to previously generated output tokens to a fixed causal window of 128 tokens.
By replacing the standard Multi-Head Attention layers of the DeepSeek OCR decoder with R-SWA, the model ensures that the KV cache remains constant throughout the entire decoding process. The visual fidelity of the original document is preserved because the model never “forgets” the reference tokens, while the output generation remains efficient because it only tracks the most recent history.
Performance and Efficiency
The architectural shift to R-SWA delivers measurable improvements. Built on the highly efficient DeepEncoder, which compresses images at a 16x rate, Unlimited OCR manages the initial visual footprint effectively.
When evaluated on the OmniDocBench v1.5 benchmark, the model achieved a 93.23% score, marking a 6% overall improvement over the baseline DeepSeek OCR. Perhaps more importantly, the efficiency metrics demonstrate that the model avoids the performance degradation seen in standard transformers. The inference speed hits 5580 tokens per second, a 12.7% increase over the baseline, and this speed remains consistent regardless of whether the model is parsing one page or forty pages.
Takeaway
Unlimited OCR proves that we can achieve one-shot, long-horizon document parsing without compromising on accuracy or speed. By implementing R-SWA, we can keep the KV cache constant and effectively simulate human working memory. While there are still limits imposed by the initial prefill length, the current results suggest that this approach is highly scalable and holds significant potential for other reference-based tasks like automatic speech recognition and machine translation.