Version 5.0.0 · Apache-2.0 · C++ and Python

Holoscan Inference

Holoscan Inference runs TensorRT models inside your Holoscan graph. Give InferenceOp one or more models and the tensors each one consumes and produces. It builds or loads a cached TensorRT engine per model and picks the fastest dispatch path that is safe for your configuration. TensorPreprocessorOp turns camera frames into the fixed-shape tensors a model expects, with every step on the GPU.

At a glance

Package	`holoscan-inference`
Version	5.0.0
C++ namespace	`holoscan::ops`
Python namespace	`holoscan.inference`
Holoscan SDK	5.0.0 or later, tested with 5.0.0
Runtime	TensorRT 10 and CUDA 13
Platforms	x86_64, aarch64
Languages	C++, Python
License	Apache-2.0
Maintainer	Holoscan Team, NVIDIA
Debian package	`holoscan-inference`
PyPI package	`holoscan-inference-cu13`

Operators

Operator	What it does
`InferenceOp`	Runs one or more TensorRT models from `.onnx` sources or prebuilt `.trt` engines. Declares one port per configured tensor and chooses a dispatch path automatically.
`TensorPreprocessorOp`	Converts RGB24, YUYV, NV12, or I420 frames into resized, normalized RGB tensors in `NCHW` or `NHWC` layout, using NPP on the Holoscan CUDA stream.

InferenceOp is backed by HoloInfer, the module’s TensorRT toolkit. The first run builds an engine next to each .onnx file and later runs reuse it. The cached filename encodes the GPU, the TensorRT version, and every build setting, so a changed configuration always triggers a rebuild.

Dispatch without tuning

InferenceOp picks one of seven dispatch modes when it starts, based on the configuration you already gave it. There is no opt-in.

Mode	Models	Shapes	Concurrency
`kFast`	1	static	—
`kSeqFast`	N	static	sequential
`kParFast`	N	static	parallel, one CUDA stream per model
`kDynFast`	1	dynamic	—
`kDynSeqFast`	N	dynamic	sequential
`kDynParFast`	N	dynamic	parallel
`kStandard`	N	any	work queue, always available

The fast modes need TensorRT models, device-resident inputs, and no chained models. When a setting rules out a faster mode, InferenceOp uses a safe one and logs the reason with a suggested fix. CUDA Graphs are on by default. Add a green-context pool to give each concurrent model its own SM partition.

Key parameters

Parameter	Default	What it does
`model_path_map`	required	Model name to `.onnx` or `.trt` path
`input_map` / `output_map`	required	Model name to ordered input or output tensor names
`is_engine_path`	`false`	Treat paths as prebuilt TensorRT engines
`enable_fp16`	`false`	Build FP16 engines on TensorRT 10
`dla_core`	`-1`	DLA core to use; `-1` disables DLA
`parallel_inference`	`true`	Run independent models in parallel when possible
`dynamic_input_dims`	`false`	Allow input shapes to change between ticks
`enable_cuda_graphs`	`true`	Capture and replay TensorRT work with CUDA Graphs
`input_on_cuda` / `output_on_cuda` / `transmit_on_cuda`	`true`	Keep inputs, internal outputs, and emitted tensors on the GPU
`cuda_stream_pool` / `cuda_green_context_pool`	none	Dedicated streams and SM partitions for concurrent models

Load the same fields from YAML with InferenceOpParams::from_yaml(path) for one operator per file, or from_yaml_node(node, config_dir) when one file configures several operators.

Examples

The module includes eight example configurations, from a single model to concurrent dynamic-shape models with CUDA Graphs and isolation.

Configuration	Scenario
`01_single_model`	One TensorRT model on GPU-resident tensors
`02_multi_model_independent`	Two independent models, run sequentially
`03_multi_model_chained`	One model’s output feeds another model’s input
`04_multi_model_concurrent`	Two independent models on dedicated streams
`05_multi_model_concurrent_graphs`	Concurrent models with CUDA Graphs
`06_multi_model_concurrent_isolated`	Concurrent models with a green-context pool
`07_dynamic_shape_single`	One model with changing input dimensions
`08_dynamic_shape_concurrent_isolated`	Dynamic shapes, concurrency, CUDA Graphs, and isolation together

Use it in an application

Link the installed CMake package:

find_package(holoscan-inference REQUIRED)
target_link_libraries(my_pipeline PRIVATE holoscan::inference)

holoscan::inference links both operators. Individual targets are available for selective linking.

From Python, install holoscan-inference-cu13 and import the operator:

import holoscan as hs
from holoscan_inference.operators import InferenceOp

graph = hs.Graph("inference")
inference = InferenceOp(
    graph,
    "inference",
    model_path_map={"detector": "/models/detector.onnx"},
    input_map={"detector": ["frames"]},
    output_map={"detector": ["detections"]},
)

Importing holoscan alone does not load TensorRT. The inference package loads it only when you import it.

To prepare camera frames for a model, put TensorPreprocessorOp in front of InferenceOp. For NV12 or I420 camera output from Holoscan Camera, set input_kind: image so the plane layout travels with each frame.

Source is in the Holoscan SDK repository.

All modules