|

NVIDIA Releases TensorRT Model Connect in Public Preview: Hugging Face Checkpoint to Native C++ Inference in Two Commands

ByRicardo

NVIDIA has launched TensorRT Model Connect (TRTMC) in public preview, an open-source mission that takes a supported Hugging Face or native checkpoint to end-to-end TensorRT inference in two instructions. There is no intermediate ONNX export step. The construct produces a versioned .bundle artifact that runs via native C++ process APIs, so inference can execute in a C++ service, embedded software, or robotics stack with out PyTorch in the runtime path. The mission is Apache-2.0 licensed and ships as a group of family-owned reference implementations moderately than a single generic converter. NVIDIA additionally states that your complete mission — mannequin implementations, efficiency tuning, assessments, integrations, and docs — was constructed utilizing OpenAI Codex brokers underneath human path and assessment.

Is it deployable?

Yes, for analysis and native integration work, with actual situations. The code is open and installable. Release wheels presently goal Linux aarch64 solely, with Python 3.10 or 3.12, glibc 2.39 or newer, and TensorRT 11.1.0.106. x86_64 wheels will not be revealed; x86_64 customers should take the Docker source-build path.

  • Company stage: Best match right this moment is groups that already personal their inference stack: NVIDIA-shop startups, robotics and gadget firms, and platform or inference groups inside mid-size and enormous enterprises. Small groups transport a Python service get much less from it. Regulated enterprises ought to look forward to a tagged launch earlier than standardizing on it.
  • Industries: Robotics and autonomous machines, industrial inspection and manufacturing, automotive in-vehicle compute, medical gadgets, protection and aerospace edge techniques, and media processing — wherever inference has to stay inside a C++ binary moderately than a Python server.
  • Applications: On-device textual content technology, speech recognition and synthesis, OCR and doc parsing, embeddings and reranking for a retrieval service written in C++, diffusion picture and video technology, segmentation, and time-series forecasting.

The two instructions

The fast begin builds and runs Qwen3-0.6B:

trtmc construct Qwen/Qwen3-0.6B --precision bf16 --max-cache-length 16384 --output qwen3-0.6b.bundle
trtmc run ./qwen3-0.6b.bundle --prompt "What is the capital of France? Answer in one phrase." --chat-template --no-thinking

The identical .bundle masses from C++ with trtmc::load("./qwen3-0.6b.bundle").

The bundle is the precise design choice

TRTMC splits construct and runtime at a versioned artifact. Python owns checkpoint decision and TensorRT engine building. Native profiles then execute inference in C++ with out PyTorch. A small variety of hybrid profiles invoke a helper Python executable, and their manifests declare that dependency explicitly.

Applications name process APIs — generate(), transcribe(), generate_image(), embed(), resolve() — as an alternative of sustaining conversion levels and per-model software glue. trtmc examine exposes bundle form, mannequin household, precision, runtime id, and engines, which makes the artifact auditable moderately than opaque.

NVIDIA frames the traditional route as PyTorch → ONNX or TorchScript → TensorRT → model-specific C++ integration, and names the failure modes it removes: export gaps, repeated per-model integration, and validation unfold throughout a number of conversion artifacts.


Key Takeaways

  • Two instructions take a supported Hugging Face checkpoint to native C++ TensorRT inference, with no ONNX step.
  • A versioned .bundle is the handoff between the Python construct and a PyTorch-free C++ runtime.
  • The July 29, 2026 GB300 snapshot covers 105 profiles throughout 76 households; 102 beat their declared reference by greater than 5%.
  • Wheels are Linux aarch64 solely right this moment; x86_64 requires the Docker supply construct.

Check out the GitHub Repo. Also, be at liberty to observe us on Twitter and don’t overlook to be part of our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to accomplice with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and many others.? Connect with us

The submit NVIDIA Releases TensorRT Model Connect in Public Preview: Hugging Face Checkpoint to Native C++ Inference in Two Commands appeared first on MarkTechPost.

Similar Posts