Most models see a person as a shape. Meta's new Sapiens2 tries to see them as matter.
Computer vision has a nasty habit of confusing appearance with identity. A model trained to ignore lighting changes—a useful trick for spotting a person in any condition—ends up forgetting what skin actually looks like. Meta AI calls this representation drift.
Their fix, Sapiens2, is a model that handles human pose and segmentation while also recovering a surface's true color, or albedo. It does this by refusing to choose between two opposing goals: being blind to distracting light, and seeing color accurately.
Its aggressive augmentation strategies like color jitter, blurring, can strip away appearance cues like skin tone or lighting conditions that are critical for tasks like albedo estimation (recovering the true color of a surface independent of lighting). This is what the research team calls representation drift.
Sapiens2 addresses this problem directly by combining both objectives: a masked image reconstruction loss (LMAE) to preserve low-level fidelity, and a global contrastive loss (LCL) on the [CLS] token using a student-teacher framework based on DINOv3, where the teacher’s parameters are an exponential moving average (EMA) of the student. Crucially, color augmentations are not applied to global views used for the MAE objective, preserving the appearance cues needed for photorealistic tasks. The joint objective is L = LMAE + λLCL.
https://arxiv.org/pdf/2604.21681
The Data: Humans-1B
Getting 1 billion training images right required a multi-stage filtering pipeline. Starting from a web-scale pool of approximately 4 billion images, Meta team applied bounding box detection, head-pose estimation, aesthetic and realism scoring, CLIP-based feature filtering, and text-overlay detection. The result is a curated corpus where every image contains at least one prominent person with a minimum short-side resolution of 384 pixels.
To ensure diversity, the research team used perceptual hashing and deep-feature nearest-neighbor pruning for deduplication, then clustered visual embeddings and applied selective sampling to balance the dataset across poses, viewpoints, occlusion levels, clothing types, and lighting conditions. No task labels or human-specific priors were injected during pretraining -- just images.
The Architecture: Scaling to 5B and 4K
Sapiens2 introduces four model sizes: 0.4B, 0.8B, 1B, and 5B parameters, each at native 1K resolution.
The technical heart of it is a balance. One part of the model rebuilds masked patches of an image, which needs accurate color. Another part learns a global, abstract representation of the scene, which should ignore color tricks.
Sapiens2 lets each part do its job without interference. It's a simple idea that requires precise engineering.
That engineering extends to the data. The one billion images it trained on weren't just scraped. They were filtered from four billion raw candidates.
Every image was checked for quality, deduplicated, and balanced for pose and lighting. The goal was a dataset of pure, varied human presence, with no hand-holding from human-applied labels. The model comes in sizes from 0.4 to 5 billion parameters, all working at high resolution.
This isn't about throwing more data at a problem. It's about building a model that understands a person is more than a silhouette. They have a surface.
And that surface has a true color, regardless of the shadows falling on it.
How does Sapiens2 address the challenge of representation drift in computer vision?
Sapiens2 tackles representation drift by using a masked image reconstruction loss (LMAE) that preserves critical appearance cues like skin tone and lighting conditions. This approach helps prevent the loss of important visual information during model training, particularly for complex tasks like albedo estimation.
What unique capabilities does Sapiens2 offer for human-focused visual tasks?
Sapiens2 provides a comprehensive solution for multiple visual tasks, including pose detection, semantic segmentation, surface normals, point-cloud mapping, and albedo recovery. The model is designed to handle high-resolution images and can potentially capture fine details like tooth and gum separation and precise finger motion tracking.
What makes Sapiens2 different from existing specialized neural networks?
Unlike traditional patchwork approaches that use multiple specialized networks, Sapiens2 offers a one-stop solution for various human-focused visual tasks. By training on high-resolution data and implementing advanced augmentation strategies, the model aims to provide a more integrated and versatile approach to computer vision challenges.
We use cookies to analyze site traffic and improve your experience. By clicking "Accept", you consent to our use of cookies.
Learn more about our privacy policy
Open Source 
Open Source 
Open Source 
Related Topic 
Related Topic 
From Archives 
From Archives 
From Archives 
From Archives 