Editorial illustration for H Company's NeoMME Encoders Drop Vision Tower, Boost Masked-Token Accuracy by Over 38 Points
NeoMME Encoders Drop Vision Tower, Boost Accuracy 38%
Nearly every visual document retriever running in production right now is built on a repurposed generative model. ColPali started the trend, and everything that followed inherited its structure: a vision-language model stripped for parts, still hauling around a separately pretrained vision tower and a causal decoder that sits there doing nothing since retrieval never generates a single token. That's dead weight on both the parameter count and the compute bill for a job that only needs good representations, not text generation.
H Company is betting that the whole approach is backwards. Its new release, NeoMME, comes in 260M and 800M parameter sizes, and it throws out the vision tower and the decoder entirely. Instead, one bidirectional Transformer handles multilingual text tokens and raw 32×32 RGB image patches through the identical layers, trained from random initialization rather than inherited from an existing model.
The retrieval-tuned version, NeoMME-Retriever, posts 0.523 nDCG@10 on ViDoRe v3 at just 260M parameters. H Company has released every checkpoint under Apache 2.0 with support already built into Hugging Face Transformers.
H Company has released NeoMME, a family of 260M and 800M bidirectional encoders that drops both components. One Transformer processes multilingual text tokens and raw 32×32 RGB image patches through the same layers, trained from random initialization. The retrieval fine-tune, NeoMME-Retriever, reaches 0.523 nDCG@10 on ViDoRe v3 at 260M parameters.
Why this matters
NeoMME is a bet that the ColPali lineage got the architecture backwards. Bolting a retrieval head onto a generative vision-language model made sense when nobody wanted to train encoders from scratch, but it left production systems carrying a causal decoder that never decodes anything, plus a vision tower trained for a different job entirely. H Company's numbers, 38.4 and 40.5 points of masked-token accuracy gained just from seeing page patches at 90% masking, suggest that overhead was masking a real capability gap, not just wasting FLOPs.
For teams running visual document retrieval at scale, that's the part worth testing directly: does a 260M single-tower encoder trained on 524 billion packed tokens actually beat a repurposed 7B-plus VLM encoder on your retrieval benchmarks, not just on paper. We'd want to see head-to-head latency and cost numbers before treating this as a drop-in replacement. But the direction, purpose-built bidirectional encoders instead of recycled decoders, is the more defensible engineering position, and it's worth watching whether ColPali's successors start dropping their vision towers too.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives Researchers Propose "AI Psychosis" Diagnosis to Speed Treatment
From Archives Google adds AI music creation to Gemini with Lyria 3.5
Common Questions Answered
What architectural components does NeoMME eliminate compared to ColPali-based retrievers?
NeoMME removes both the vision tower and the causal decoder that are present in ColPali-based visual document retrievers. These components were unnecessary dead weight since document retrieval tasks don't require token generation, only efficient representations. By eliminating them, NeoMME reduces parameter count and computational costs while improving performance.
How does NeoMME process both text and image data in its unified architecture?
NeoMME uses a single Transformer that processes both multilingual text tokens and raw 32×32 RGB image patches through the same layers, trained from random initialization. This unified bidirectional encoder approach eliminates the need for separate vision and language components, allowing efficient joint processing of multimodal content.
What performance improvements does NeoMME-Retriever achieve on the ViDoRe v3 benchmark?
NeoMME-Retriever reaches 0.523 nDCG@10 on ViDoRe v3 at 260M parameters, while gaining 38.4 to 40.5 points of masked-token accuracy improvement through aggressive masking of page patches at 90% masking rates. These substantial gains demonstrate that the encoder-only architecture is better suited for retrieval tasks than repurposed generative models.
Why does H Company argue that the ColPali architecture was suboptimal for production document retrieval systems?
H Company contends that bolting a retrieval head onto generative vision-language models left production systems carrying unnecessary components: a causal decoder that never generates tokens and a vision tower trained for a different task. This architectural mismatch resulted in inefficient parameter usage and higher computational costs for a job that only requires good representations, not generation capabilities.
What are the available parameter sizes for H Company's NeoMME encoder family?
H Company released NeoMME as a family of bidirectional encoders in two sizes: 260M and 800M parameters. Both models use the same unified architecture for processing text and image patches, allowing users to choose between efficiency and performance based on their deployment requirements.
Further Reading
- NeoMME: an efficient Multimodal-native and Multilingual foundation encoder - Hugging Face Blog
- NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder - Hugging Face Papers
- ColPali: Efficient Document Retrieval with Vision Language Models - arXiv
- Late Interaction & Efficient Multi-modal Retrievers Need More Than Just a Vector Index - LanceDB Blog
