Editorial illustration for Know3D uses image model with Qwen2.5‑VL to edit hidden sides of 3D objects
Know3D AI Reveals Hidden Sides of 3D Object Editing
Know3D uses image model with Qwen2.5‑VL to edit hidden sides of 3D objects
Staring at a 3D model on screen is a fundamentally frustrating act. You can only edit what you can see. The back of the object?
It's a blind spot. A new system, Know3D, aims to fix that by letting artists use a simple text prompt to reshape the hidden geometry. The obvious approach—shoving a large language model directly into a 3D generator—fails.
They operate in completely different languages. The team's solution was to find a translator.
So Know3D takes a detour, slotting an image generation model between the language model and the 3D generator to act as a translator. The setup uses Qwen2.5-VL as the language model, Qwen-Image-Edit for image generation, and Microsoft's Trellis.2 as the 3D generator. The language model reads the text instruction and analyzes the input image.
The image generator then turns that understanding into spatial-structural information that steers the 3D generator. The trick is figuring out what information to pull from the image generator. The team tested three options: an internal image representation grabbed right before the final output, image features extracted from it via Meta's DINOv3, and the model's internal intermediate states during generation.
The last option won by a clear margin--these intermediate states carry both semantic and spatial information without relying on pixel-level accuracy or mistakes in the final image.
Those intermediate states are everything. They capture an object's meaning and spatial structure long before the image model commits to—and often botches—the final pixels. That's the real breakthrough here.
The system treats Qwen-Image-Edit not as a final artist but as a conceptual sketchpad, where its messy, unfinished thoughts prove far more valuable than a polished, pixel-perfect lie. The outcome feels less like algorithmic guesswork. It becomes precise internal sculpting.
You truly don't have to see it to shape it.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives Zhipu AI's GLM-5V-Turbo converts mockups to web code, tops coding and GUI benchmarks
From Archives Anthropic adds desktop control to Claude via Claude Code and Cowork on Mac and Windows
Common Questions Answered
How does Know3D solve the challenge of editing hidden sides of 3D objects?
Know3D introduces an innovative approach by inserting an image generation model between the language model and 3D generator. The system uses Qwen2.5-VL to understand text instructions, Qwen-Image-Edit to generate spatial-structural information about unseen surfaces, and Microsoft's Trellis.2 to integrate these suggestions into the 3D model.
What specific models are used in the Know3D pipeline for 3D object editing?
The Know3D system utilizes three key models: Qwen2.5-VL as the language model to interpret text instructions, Qwen-Image-Edit for generating image-based spatial suggestions, and Microsoft's Trellis.2 as the 3D generator to incorporate those suggestions into the final 3D object. This multi-model approach allows for more sophisticated editing of previously unseen object surfaces.
Why is editing hidden sides of 3D models traditionally difficult?
Traditional 3D modeling pipelines struggle to edit surfaces that are not visible in the initial rendering, forcing creators to manually sculpt back sides or accept generic approximations. This limitation significantly slows down the creative process and requires advanced technical skills to overcome.
Further Reading
- Know3D: Prompting 3D Generation with Knowledge from Vision Language Models — arXiv
- Qwen2.5 VL Official Release — Qwen Official Blog
- Use Qwen2.5-VL for Zero-Shot Object Detection — Roboflow Blog
- Qwen2.5-VL: A hands on code walkthrough — Towards AI
- Qwen Image Edit Full Tutorial: 26 Different Demo Cases, Prompts — YouTube (secourses)
