Editorial illustration for Researchers Generate 3D Assets From Single Images With Relightable Gaussians
3D Assets From Single Images With Relightable Gaussians
A team of ten researchers, including Mayank Singh, Michele Stoppa and David E. Jacobs, has built a system called Luce that turns a single photo into a 3D object you can light from any angle. The problem they're tackling is a familiar one in computer graphics: most image-to-3D tools spit out a shape that looks fine under the lighting it was captured in, then falls apart the moment you drop it into a game engine or a rendering pipeline with different lights. Luce gets around that by generating not just geometry but the full set of physically based rendering data, albedo, metallic-roughness values and surface normals, that let an asset respond correctly to new light sources.
The core idea is a voxelized cloud of 3D Gaussians, with separate primitives assigned to each material property instead of cramming everything into one generic point. That separation is what lets the model hold onto fine detail, things like printed text, logos or engraved lettering, while still producing something that behaves like a real object under a spotlight or in shadow. The work sits alongside related projects on Gaussian-based head reconstruction and multiview diffusion, part of a broader push to make generated 3D assets usable outside a lab demo.
High-fidelity image-to-3D generation requires a 3D representation that captures both geometry and appearance. To support relighting and integration into standard rendering pipelines, the representation should include physically based rendering (PBR) modalities such as albedo, metallic-roughness, and surface normals.
Why this matters
Luce tackles a problem that's dogged image-to-3D pipelines for years: geometry without usable materials. Most single-image reconstruction tools hand you a shape baked under whatever lighting the source photo had, which means the asset looks wrong the moment you drop it into a different scene. By folding relightability into the Gaussian representation itself, and compressing everything through a variational autoencoder into one material-aware latent space, the ten authors from this team are betting that generation and shading shouldn't be separate steps bolted together in post-production. The rectified-flow transformer doing the actual generation, conditioned on multi-layer features from a pretrained encoder, suggests they're leaning on existing 2D vision backbones rather than training 3D understanding from scratch.
For anyone building game assets, product visualizations, or AR content, the real test is whether Luce's outputs hold up under lighting conditions nothing like the training photo. Papers claiming material-aware latents are common; deployable relighting that survives contact with an actual rendering pipeline is rarer. Worth watching for released code or benchmark comparisons against existing Gaussian splatting methods before assuming this is production-ready.
Related Reading
Policy & Regulation After ditching AI fitness apps and a Fitbit, I return to Peloton classes
Policy & Regulation Grammarly faces class-action suit over AI ‘Expert Review’ feature
Policy & Regulation Akamai Proposes SAFE Cybersecurity Guidelines for AI Threats
From Archives Watermark Alters AI's Internal Probability Numbers
From Archives Meta AI Glasses Sales Surge Raises Covert Recording Concerns
Common Questions Answered
What problem does the Luce system solve in image-to-3D generation?
Luce addresses the issue where most image-to-3D tools produce 3D objects that only look correct under the original lighting conditions from the source photo. When these assets are placed into different rendering pipelines or game engines with different lighting, they appear incorrect and unusable. Luce solves this by generating relightable 3D assets that maintain visual fidelity under any lighting condition.
What physically based rendering modalities does Luce include in its 3D representation?
Luce generates 3D assets that include physically based rendering (PBR) modalities such as albedo, metallic-roughness, and surface normals. These PBR components enable the generated 3D objects to be properly integrated into standard rendering pipelines and respond realistically to different lighting conditions. This approach ensures that the 3D assets maintain their visual quality across various rendering environments.
How does Luce compress its material-aware 3D representation?
Luce uses a variational autoencoder to compress its material-aware Gaussian representation into a single latent space. This compression technique allows the system to efficiently encode both geometry and appearance information while maintaining the relighting capabilities necessary for integration into different rendering pipelines. The variational autoencoder approach enables efficient storage and manipulation of the 3D assets.
What is the key advantage of folding relightability into Luce's Gaussian representation?
By incorporating relightability directly into the Gaussian representation itself, Luce ensures that 3D assets generated from single images can be realistically rendered under any lighting condition. This eliminates the common problem where image-to-3D tools produce geometry that is baked under specific lighting and fails to adapt to new rendering environments. The result is high-fidelity 3D assets that are immediately usable in game engines and professional rendering pipelines.
Further Reading
- Luce: Relightable Gaussians for 3D Asset Generation - arXiv
- Relightable 3D Gaussians: Realistic Point Cloud Relighting with BRDF Decomposition and Ray Tracing - arXiv
- Rng: Relightable Neural Gaussians - arXiv
- SU-RGS: Relightable 3D Gaussian Splatting from Sparse Views under Unconstrained Illumination - ICCV 2025
- GaRe: Relightable 3D Gaussian Splatting for Outdoor Scenes from Unconstrained Photo Collections - arXiv
