Editorial illustration for C++ and TensorRT Samples Show Local AI App Development Path
NVIDIA's C++ Samples Simplify Local AI Development
C++ and TensorRT Samples Show Local AI App Development Path
NVIDIA has published an open-source collection of C++ samples called Do Inference Now Deploy, aimed at a specific gap in local AI development: getting a trained model off a checkpoint and into a working application without dragging in a model-specific runtime. The project pairs ONNX Runtime with the TensorRT RTX execution provider, targeting both Windows and Linux, with the same API reachable through WinML 2.0.
The structure is deliberately split. A Python exporter handles the first half of the job, pulling a checkpoint from Hugging Face and converting it to ONNX. From there, a native C++ command-line application takes over, built on ONNX Runtime's session and tensor APIs. NVIDIA designed it so CUDA-specific code only shows up in optional accelerated paths, not in the shared code that any compatible execution provider can run.
That separation of concerns, export versus deployment, is the core idea behind the samples. CMake presets cover Windows and Linux builds, including Arm64, though DirectX support is limited to Windows. One sample, built around FLUX.2, leans on a newer ONNX Runtime feature for handling graphics interop directly during preprocessing and postprocessing.
Do Inference Now (DIN) Deploy is an open-source collection of practical C++ samples that bridges that gap. It combines ONNX Runtime with the NVIDIA TensorRT RTX execution provider to help developers move from a model checkpoint to a native, hardware-accelerated application on Windows and Linux.
Why this matters
For developers trying to ship AI features without dragging users through a CUDA install or locking them to one GPU vendor, DIN Deploy's approach is worth watching. Keeping the bulk of sample code in standard ONNX Runtime session and tensor APIs, with CUDA and vendor-specific kernels confined to optional accelerated paths, is a practical bet on portability over raw speed. It means the same C++ codebase can run on an execution provider that supports the required tensor APIs, whether that's TensorRT RTX, another provider, or WinML, without a rewrite.
For founders scoping local-AI products, that lowers the cost of supporting both Windows and Linux from day one. Researchers benchmarking inference across hardware get a cleaner baseline to test against. We'd flag the obvious tension: optional accelerated paths still mean someone has to maintain the CUDA-specific branches, and "runs everywhere" samples don't guarantee "runs fast everywhere." Still, open-sourcing the plumbing between checkpoint and native app is the less glamorous work that actually determines whether local AI apps become routine or stay a demo.
Related Reading
AI Tools & Apps Claude gains shared context in Excel, PowerPoint; Microsoft adds Copilot Cowork
AI Tools & Apps OpenAI Developer: AI Agent "Swarms" Waste Tokens, Create "Coordination Tax
AI Tools & Apps LG's recent webOS update adds Microsoft Copilot app, now removable
From Archives NVIDIA DOCA Agent Skills Speed BlueField App Development
From Archives The Download: An Interactive Map of 8,000 Global Castles and Fortresses
Common Questions Answered
What is Do Inference Now Deploy and what problem does it solve?
Do Inference Now Deploy (DIN Deploy) is an open-source collection of C++ samples published by NVIDIA that bridges the gap between having a trained model checkpoint and deploying it as a working application. It eliminates the need for model-specific runtimes, allowing developers to move directly from checkpoint to native, hardware-accelerated applications on Windows and Linux.
How does DIN Deploy combine ONNX Runtime with TensorRT RTX?
DIN Deploy pairs ONNX Runtime with the NVIDIA TensorRT RTX execution provider to enable hardware acceleration for AI models. This combination allows developers to access the same API through both standard ONNX Runtime and WinML 2.0, providing flexibility across different platforms and execution environments.
What is the advantage of keeping CUDA and vendor-specific kernels optional in DIN Deploy?
By confining CUDA and vendor-specific kernels to optional accelerated paths while maintaining standard ONNX Runtime session and tensor APIs, DIN Deploy prioritizes portability over raw speed. This approach allows the same C++ codebase to run on any execution provider that supports the required tensor APIs, avoiding vendor lock-in and reducing deployment friction.
Why is DIN Deploy significant for developers shipping AI features locally?
DIN Deploy is valuable for developers who want to deploy AI features without requiring users to install CUDA or locking them to a specific GPU vendor. It provides a practical solution for creating portable, hardware-accelerated applications that can run across different hardware configurations while maintaining code compatibility.
What platforms and APIs does DIN Deploy support?
DIN Deploy targets both Windows and Linux platforms and provides access to the same API through WinML 2.0. The project uses a deliberately split structure where a Python exporter handles model conversion, making it compatible with various execution providers that support standard tensor APIs.
Further Reading
- Build Local AI Apps with C++ and NVIDIA TensorRT RTX Samples - NVIDIA Technical Blog
- AI Native by Design: Lessons Learned from Building NVIDIA TensorRT Model Connect - NVIDIA Technical Blog
- Building and Running C++ Samples - NVIDIA TensorRT Documentation
- Sample Support Guide - NVIDIA TensorRT Documentation
- TensorRT - NGC Catalog - NVIDIA NGC Catalog
