DAQIRI Bare-Metal CMake Build

Bare-Metal CMake Build

This tutorial walks you through a complete bare-metal DAQIRI build on a Linux host, from verifying the kernel, driver, NIC, and CUDA prerequisites, through building a patched DPDK from source, configuring DAQIRI with CMake, installing the library, and recovering from the most common failure modes.

It is the long-form companion to the five-line cmake snippet in Getting Started.

Prefer the container build

The container build ships a dmabuf-patched DPDK and all user-space dependencies pre-installed. It is the recommended path for most users. Use this bare-metal tutorial when:

your host distribution or CUDA stack is fixed and you cannot run a container;
you are packaging DAQIRI into another product that already provides a runtime image;
you are debugging a build problem inside the container's daqiri-build stage and need to reproduce it on the host.

If none of those apply, follow System Configuration and then Raw Ethernet Benchmarking instead.

Prerequisite verification

Before installing anything, run the checks below and resolve any failures. They all run as a regular user.

Check	Command	Expected
NVIDIA driver + GPU	`nvidia-smi`	Driver version printed, one or more GPUs listed
CUDA Toolkit (11.7+)	`nvcc --version`	`Cuda compilation tools, release 11.7` or later
NIC kernel drivers	`lsmod \\| grep ib_core`	`ib_core` listed
NIC user-space tools	`ibv_devinfo`	At least one `mlx5_*` HCA in `PORT_ACTIVE` state
ConnectX device on PCIe	`lspci -d 15b3:`	One or more Mellanox/NVIDIA networking entries
Kernel dma-buf support	`zgrep CONFIG_DMA_SHARED_BUFFER /proc/config.gz`	`=y` (built-in)

nvidia-smi: command not found

Install the NVIDIA driver. On Ubuntu the simplest path is sudo ubuntu-drivers install. The driver must be a recent branch with GPUDirect support. See the system requirements table.

nvcc: command not found (but nvidia-smi works)

The runtime driver is installed but the CUDA Toolkit is not. Add the CUDA APT repository for your distribution from NVIDIA's CUDA downloads page and sudo apt install cuda-toolkit. Then put nvcc on PATH (typically /usr/local/cuda/bin).

lsmod | grep ib_core is empty

The NIC kernel drivers are not loaded. Follow Check your NIC drivers in System Configuration to install doca-ofed (or the inbox rdma-core stack) and reboot.

ibv_devinfo prints libvmw_pvrdma-rdmav34.so warning

couldn't load driver 'libvmw_pvrdma-rdmav34.so' is harmless. It refers to a VMware paravirtual RDMA userspace provider that is not relevant on bare-metal systems. As long as ibv_devinfo still lists your mlx5_* HCAs after the warning, you can ignore it. See System Configuration → Warning about libvmw_pvrdma-rdmav34.so for more context.

/proc/config.gz does not exist

Some kernels don't expose /proc/config.gz (notably the IGX/Tegra kernel). Try grep CONFIG_DMA_SHARED_BUFFER /boot/config-$(uname -r) instead. The expected value is =y. Without dma-buf support the patched DPDK built in Step 3 cannot register GPU memory regions and DAQIRI will fall back to the legacy peermem path.

Step 1: Configure the DOCA APT repository

DAQIRI's RDMA/ibverbs dependencies (libibverbs-dev, librdmacm-dev, libmlx5-1, mlnx-ofed-kernel-utils, mft) come from the DOCA repository. Run the tab that matches your platform.

IGX OS 1.1SBSA (Ubuntu 22.04)x86_64 (Ubuntu 22.04)x86_64 (Ubuntu 24.04)

export DOCA_URL="https://linux.mellanox.com/public/repo/doca/2.8.0/ubuntu22.04/arm64-sbsa/"
wget -qO- https://linux.mellanox.com/public/repo/doca/GPG-KEY-Mellanox.pub | gpg --dearmor - | sudo tee /etc/apt/trusted.gpg.d/GPG-KEY-Mellanox.pub > /dev/null
echo "deb [signed-by=/etc/apt/trusted.gpg.d/GPG-KEY-Mellanox.pub] $DOCA_URL ./"  | sudo tee /etc/apt/sources.list.d/doca.list > /dev/null

sudo apt update

export DOCA_URL="https://linux.mellanox.com/public/repo/doca/2.8.0/ubuntu22.04/arm64-sbsa/"
wget -qO- https://linux.mellanox.com/public/repo/doca/GPG-KEY-Mellanox.pub | gpg --dearmor - | sudo tee /etc/apt/trusted.gpg.d/GPG-KEY-Mellanox.pub > /dev/null
echo "deb [signed-by=/etc/apt/trusted.gpg.d/GPG-KEY-Mellanox.pub] $DOCA_URL ./"  | sudo tee /etc/apt/sources.list.d/doca.list > /dev/null

# Also need the CUDA repository: https://developer.nvidia.com/cuda-downloads?target_os=Linux
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/sbsa/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb

sudo apt update

export DOCA_URL="https://linux.mellanox.com/public/repo/doca/2.8.0/ubuntu22.04/x86_64/"
wget -qO- https://linux.mellanox.com/public/repo/doca/GPG-KEY-Mellanox.pub | gpg --dearmor - | sudo tee /etc/apt/trusted.gpg.d/GPG-KEY-Mellanox.pub > /dev/null
echo "deb [signed-by=/etc/apt/trusted.gpg.d/GPG-KEY-Mellanox.pub] $DOCA_URL ./"  | sudo tee /etc/apt/sources.list.d/doca.list > /dev/null

# Also need the CUDA repository: https://developer.nvidia.com/cuda-downloads?target_os=Linux
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb

sudo apt update

export DOCA_URL="https://linux.mellanox.com/public/repo/doca/3.2.1/ubuntu24.04/x86_64/"
wget -qO- https://linux.mellanox.com/public/repo/doca/GPG-KEY-Mellanox.pub | gpg --dearmor - | sudo tee /etc/apt/trusted.gpg.d/GPG-KEY-Mellanox.pub > /dev/null
echo "deb [signed-by=/etc/apt/trusted.gpg.d/GPG-KEY-Mellanox.pub] $DOCA_URL ./"  | sudo tee /etc/apt/sources.list.d/doca.list > /dev/null

# Also need the CUDA repository: https://developer.nvidia.com/cuda-downloads?target_os=Linux
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb

sudo apt update

Note

The container build uses DOCA 3.2.1 (see DOCA_VERSION in the Dockerfile). Earlier DOCA releases also work. The version pinning is for build reproducibility, not a strict requirement.

Step 2: Install build tooling and RDMA libraries

The package list below mirrors the base, base-deps, and dpdk stages of the Dockerfile and is the minimum needed for the recipe that follows.

sudo apt update
sudo apt install -y --no-install-recommends \
    build-essential \
    git \
    curl \
    ca-certificates \
    gnupg \
    pkg-config \
    ninja-build \
    meson \
    python3-pyelftools \
    libnuma-dev \
    libibverbs-dev \
    librdmacm-dev \
    libmlx5-1 \
    ibverbs-utils \
    mlnx-ofed-kernel-utils \
    mft

apt may select ibverbs-providers instead of libmlx5-1

On many host configurations (always on Ubuntu 24.04 with DOCA 3.2.1, where libmlx5-1 has no DOCA candidate) apt resolves libmlx5-1 to the ibverbs-providers virtual package (the inbox bundle of mlx5/mlx4 user-space providers). This is fine: libmlx5.so.1 ends up installed either way and DAQIRI links against it through libibverbs. If you want the DOCA-provided build specifically, install libmlx5-1 and libmlx5-dev after the DOCA repo from Step 1 is configured.

DAQIRI's top-level CMake requires version 3.20 or newer (see CMakeLists.txt). On Ubuntu 22.04 the distribution package is too old. Install a current build from the Kitware APT repository:

OS_CODENAME=$(. /etc/os-release && echo "$VERSION_CODENAME")
KITWARE_LINE="deb [signed-by=/usr/share/keyrings/kitware-archive-keyring.gpg] https://apt.kitware.com/ubuntu/ ${OS_CODENAME} main"

curl -fsSL https://apt.kitware.com/keys/kitware-archive-latest.asc \
    | sudo gpg --dearmor -o /usr/share/keyrings/kitware-archive-keyring.gpg
echo "${KITWARE_LINE}" | sudo tee /etc/apt/sources.list.d/kitware.list

sudo apt update
sudo apt install -y --no-install-recommends kitware-archive-keyring cmake cmake-data
cmake --version   # expect >= 3.20

Copy/paste safety

The KITWARE_LINE shell variable above (and the DISABLE_DRIVERS variable in Step 3.3) keep long single-line strings off the terminal so a copy/paste that wraps a line break into the middle of the string does not silently truncate the argument. If you paste the literal echo ... or meson setup ... commands instead, verify the resulting file/command actually contains what you intended before continuing.

Optional: Python bindings

Skip this if you do not need import daqiri from Python. To build the pybind11 bindings later with -DDAQIRI_BUILD_PYTHON=ON:

sudo apt install -y --no-install-recommends python3-dev pybind11-dev

Optional: GDS (cuFile) for device-memory file writes

Skip this unless you plan to enable -DDAQIRI_ENABLE_GDS=ON. The cuFile headers and libcufile are installed as part of the CUDA Toolkit (cuda-toolkit-* metapackage) and need no separate apt step. Runtime verification is covered in DAQIRI_ENABLE_GDS: cuFile burst writes.

Step 3: Build DPDK with DAQIRI patches

DAQIRI requires DPDK with the two patches in dpdk_patches/: dmabuf.patch enables GPUDirect through the kernel dma-buf interface (so nvidia-peermem is not required), and dpdk.nvidia.patch adds the NVIDIA-specific changes consumed by the DPDK engine. The recipe below is the literal sequence run by the dpdk stage of the Dockerfile.

3.1 Clone DAQIRI

You need the repository for dpdk_patches/ regardless of whether you also build DAQIRI from this clone in Step 4.

git clone https://github.com/NVIDIA/daqiri.git
cd daqiri

3.2 Download and patch DPDK 25.11

DPDK_VERSION=25.11
DAQIRI_REPO_DIR=$PWD   # remember the daqiri clone path before we cd away

curl -fsSL https://fast.dpdk.org/rel/dpdk-${DPDK_VERSION}.tar.xz -o /tmp/dpdk-${DPDK_VERSION}.tar.xz
tar xf /tmp/dpdk-${DPDK_VERSION}.tar.xz -C /tmp
cd /tmp/dpdk-${DPDK_VERSION}

# `git apply` is used instead of `patch -p1` because:
#   1. `dmabuf.patch` carries `.mailmap` and release-notes hunks that conflict on
#      a stock tarball, and they must be excluded. GNU `patch` 2.7.x does not
#      accept `--exclude=`; only `git apply` does.
#   2. `git apply` does not actually require the working directory to be a git
#      repo, so it works on an extracted tarball as-is.
git apply \
    --exclude=.mailmap \
    --exclude='doc/guides/rel_notes/release_26_03.rst' \
    "${DAQIRI_REPO_DIR}/dpdk_patches/dmabuf.patch"
git apply "${DAQIRI_REPO_DIR}/dpdk_patches/dpdk.nvidia.patch"

3.3 Configure, build, and install

The meson setup flags below match the container build. The disable_drivers list trims drivers DAQIRI never uses, cutting build time and the installed footprint. It is kept in a shell variable so the long comma-separated string is not at risk of being truncated by terminal copy/paste line-wrapping.

DPDK_BUILD_DIR=/tmp/dpdk-build
DPDK_INSTALL_PREFIX=/usr/local

DISABLE_DRIVERS='baseband/*,bus/ifpga/*,common/cpt,common/dpaax,common/iavf,common/octeontx,common/octeontx2,crypto/nitrox,net/ark,net/atlantic,net/avp,net/axgbe,net/bnx2x,net/bnxt,net/cxgbe,net/e1000,net/ena,net/enic,net/fm10k,net/hinic,net/hns3,net/i40e,net/ixgbe,vdpa/ifc,net/igc,net/liquidio,net/mana,net/netvsc,net/nfp,net/qede,net/sfc,net/thunderx,net/vdev_netvsc,net/vmxnet3,regex/octeontx2'

meson setup ${DPDK_BUILD_DIR} \
    --prefix=${DPDK_INSTALL_PREFIX} \
    -Dtests=false \
    -Dplatform=generic \
    -Denable_docs=false \
    -Ddisable_drivers="${DISABLE_DRIVERS}"

cd ${DPDK_BUILD_DIR}
ninja
sudo meson install
sudo ldconfig

3.4 Verify

pkg-config --modversion libdpdk   # expect 25.11.x
pkg-config --libs libdpdk         # should print -lrte_eal -lrte_mbuf ... etc.

If pkg-config cannot find libdpdk, add the install location to its search path:

export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:/usr/local/lib/x86_64-linux-gnu/pkgconfig:${PKG_CONFIG_PATH:-}

(Replace x86_64-linux-gnu with aarch64-linux-gnu on ARM hosts.)

Step 4: Configure DAQIRI with CMake

From the daqiri repository root (the clone you made in Step 3.1):

cmake -S . -B build \
    -DCMAKE_BUILD_TYPE=Release \
    -DBUILD_SHARED_LIBS=ON \
    -DDAQIRI_ENGINE="dpdk ibverbs" \
    -DDAQIRI_BUILD_EXAMPLES=ON \
    -DDAQIRI_BUILD_PYTHON=OFF

Deactivate conda before configuring DPDK

An active conda environment (including (base)) can break meson setup for DPDK in Step 3.3. Run conda deactivate before building DPDK from source.

DAQIRI itself prefers the vendored third_party/yaml-cpp submodule by default, so example benchmarks link correctly even when conda is active. Initialize submodules before configuring: git submodule update --init third_party/yaml-cpp. Set -DDAQIRI_PREFER_SYSTEM_YAML_CPP=ON only if you intentionally want the system package.

CMake options reference

Most users can start with the configure command above. Use this table when you need to enable a specific feature or adjust the build for a particular host.

Option	Default	When to change it
`CMAKE_BUILD_TYPE`	n/a	Use `Release` for benchmarks and installed builds.
`BUILD_SHARED_LIBS`	n/a	Use `ON` to produce `libdaqiri.so` and exported package metadata; use `OFF` only when you intentionally need static libraries.
`DAQIRI_ENGINE`	`"dpdk ibverbs"`	Space-separated optional engines to compile. Valid values are `dpdk` and `ibverbs`; Linux UDP/TCP sockets are always built in. Use a smaller set only when packaging a narrower build.
`DAQIRI_IBVERBS_DLOPEN`	`OFF`	Load the raw ibverbs and RoCE dependencies through versioned glue DSOs at engine initialization. Useful when statically linking DAQIRI while using shared RDMA libraries at runtime.
`DAQIRI_BUILD_EXAMPLES`	`ON`	Builds the `daqiri_bench_*` executables under `build/examples/`. Leave on for smoke tests and benchmark workflows. The `daqiri_config_validate` tool is always built and installed, including when this option is `OFF`.
`DAQIRI_BUILD_APPLICATIONS`	`OFF`	Builds heavier end-to-end examples under `applications/`, currently the TensorRT ResNet inference app. Requires the extra application dependencies.
`DAQIRI_BUILD_PYTHON`	`OFF`	Builds pybind11 Python bindings. Enable only when you need `import daqiri`.
`BUILD_TESTING`	`ON`	Builds and registers the hardware-free C++ tests under `tests/cpp/` with CTest. Set to `OFF` to omit test targets from a production-only build.
`DAQIRI_ENABLE_GDS`	`OFF`	Enables cuFile-backed burst file writes from CUDA device memory. Enable only on systems with a working GDS runtime.
`DAQIRI_ENABLE_OTEL_METRICS`	`OFF`	Enables OpenTelemetry metrics instrumentation. Applications still configure the SDK reader/exporter.
`DAQIRI_ENABLE_S3`	`OFF`	Enables AWS SDK-backed asynchronous raw packet writes to S3-compatible object stores.
`DAQIRI_REORDER_GPU_PROFILE`	`OFF`	Adds CUDA event timing around reorder kernels. Enable only while profiling.
`DAQIRI_PREFER_SYSTEM_YAML_CPP`	`OFF`	Prefer a system `yaml-cpp` instead of the vendored submodule. Keep `OFF` when a conda/miniforge environment is on `PATH`.
`CMAKE_CUDA_ARCHITECTURES`	`80;90`, plus `121` with CUDA Toolkit 13.0+	Override when your GPU is not covered by the default architecture list.

The sections below explain the options that most often need case-by-case guidance.

`DAQIRI_ENGINE`: engine selection

DAQIRI_ENGINE is a space-separated list controlling which optional engines are compiled into libdaqiri.so. Linux UDP/TCP sockets are always built in, so the only values are dpdk and ibverbs. Three recipes cover most use cases:

Recipe	What you get	When to use
`-DDAQIRI_ENGINE="dpdk"`	Raw Ethernet (DPDK) + built-in UDP/TCP sockets	No RoCE/RDMA, smallest NIC build
`-DDAQIRI_ENGINE="ibverbs"`	Raw Ethernet (ibverbs), RDMA/RoCE + built-in UDP/TCP sockets	No DPDK dependency
`-DDAQIRI_ENGINE="dpdk ibverbs"`	Raw Ethernet + RDMA/RoCE + built-in UDP/TCP sockets	Recommended default, matches the container

Sockets are always available; ibverbs backs RoCE

The socket engine (Linux UDP/TCP) is always built, so it never appears in DAQIRI_ENGINE. Its RoCE path delegates to the ibverbs engine, so roce:// endpoints work only when ibverbs is in the list. Internally the ibverbs engine lives under src/engines/rdma (target daqiri_rdma); ibverbs is just the user-facing name, chosen so an alternative RDMA engine (such as DOCA) could be added later.

`DAQIRI_IBVERBS_DLOPEN`: runtime RDMA libraries

Enable -DDAQIRI_IBVERBS_DLOPEN=ON to keep libibverbs, libmlx5, and librdmacm out of the DAQIRI engines' link dependencies. This works with both static and shared DAQIRI builds. For a static build with raw ibverbs, RoCE, and Linux UDP/TCP sockets:

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \
  -DBUILD_SHARED_LIBS=OFF -DDAQIRI_ENGINE="ibverbs" \
  -DDAQIRI_IBVERBS_DLOPEN=ON -DDAQIRI_BUILD_PYTHON=OFF
cmake --build build -j
cmake --install build --prefix /opt/daqiri

Run these build commands in the project container. The build still requires RDMA development headers and shared libraries to compile the glue libraries. Installation includes libdaqiri_raw_verbs_glue.so.1 (verbs/mlx5) and libdaqiri_cm_verbs_glue.so.1 (verbs/RDMA CM) in the library directory. Deploy these with compatible system RDMA shared libraries. The application executable can link DAQIRI archives, but is not a fully static, self-contained deployment.

DAQIRI loads only the glue needed by the selected engine, on initialization. It first checks DAQIRI_RDMA_GLUE_DIR, if set; this directory is authoritative. Otherwise it searches beside the engine DSO or executable, then uses the system dynamic loader search path. Static consumers resolve the executable's location independently of the working directory, including when launched with a relative path and initialized after changing directories. For an installed static consumer, for example:

export DAQIRI_RDMA_GLUE_DIR=/opt/daqiri/lib

Adjust the path to your installation's library directory (which may be lib64 or an architecture-specific subdirectory). The environment variable locates the DAQIRI glue libraries; their RDMA dependencies must also be discoverable by the system loader. For build-tree runs, use DAQIRI_RDMA_GLUE_DIR="$PWD/build/lib". Missing libraries, missing entry points, and incompatible glue tables fail engine initialization with a diagnostic. Successful loads remain open for the process lifetime. Linux UDP/TCP sockets and hardware-free configuration validation do not require either glue library.

The exported CMake targets and pkg-config metadata omit direct RDMA dependencies in this mode. For static CMake consumers, group the common and engine archives when required by the linker, as in the project's C++ test targets. The pkg-config --static --libs daqiri output includes that archive group.

This option applies to DAQIRI's own ibverbs/RDMA engines. A DPDK-enabled build can still depend on verbs/mlx5 through its mlx5 PMD; configure DPDK separately for runtime loading, or use DAQIRI_ENGINE="ibverbs" to omit DPDK.

`DAQIRI_BUILD_PYTHON`: pybind11 bindings

-DDAQIRI_BUILD_PYTHON=ON builds the daqiri Python module from python/. It requires the python3-dev and pybind11-dev packages from the optional step in Step 2. Leave it OFF (the default) if you only consume DAQIRI from C++.

`DAQIRI_ENABLE_GDS`: cuFile burst writes

-DDAQIRI_ENABLE_GDS=ON enables the cuFile-backed file-write path for bursts whose payload lives in CUDA device memory. The build requires cufile.h and libcufile (both shipped with the CUDA Toolkit). At runtime you also need the nvidia-fs kernel module loaded and a GDS-supported destination filesystem. Verify with:

lsmod | grep nvidia_fs
/usr/local/cuda/gds/tools/gdscheck.py -p

For regular cuFile/GDS over local NVMe, gdscheck.py -p should report NVMe : Supported, and ext4 destinations must be mounted with data=ordered or use another GDS-supported filesystem such as XFS. If nvidia-fs is not loaded, or the destination storage is not supported, DAQIRI returns NOT_SUPPORTED for CUDA device-backed burst writes. Host-backed burst writes continue to use POSIX APIs and do not require GDS.

`CMAKE_CUDA_ARCHITECTURES`: GPU compute capability

The default CUDA architectures DAQIRI compiles for are 80;90 (A100, H100), with 121 (GB10) appended automatically when the configuring CUDA compiler is 13.0 or newer. The logic lives in src/CMakeLists.txt (search for CMAKE_CUDA_ARCHITECTURES).

If your GPU is a different generation, override it on the CMake command line:

# Example: Ada only (sm_89, e.g. RTX 6000 Ada)
cmake -S . -B build ... -DCMAKE_CUDA_ARCHITECTURES=89

# Example: A100 + Ada + Hopper, skip GB10
cmake -S . -B build ... -DCMAKE_CUDA_ARCHITECTURES="80;89;90"

# Example: RTX PRO 6000 Blackwell Server/Workstation Edition only (sm_120)
cmake -S . -B build ... -DCMAKE_CUDA_ARCHITECTURES=120

The override is honored because src/CMakeLists.txt only sets CMAKE_CUDA_ARCHITECTURES when it is not already defined, so a user-supplied -D value takes precedence. Common values:

GPU family	Architecture
A100	`80`
RTX 30xx, A40	`86`
RTX 40xx, RTX 6000 Ada, L40	`89`
H100, H200	`90`
Blackwell datacenter (B100, B200, GB100, GB200)	`100`
Blackwell consumer / workstation (RTX PRO 6000 Blackwell, RTX 5090)	`120`
GB10 (DGX Spark)	`121`

Default list omits sm_86, sm_89, sm_100, and sm_120

If your GPU is Ampere (RTX 30xx, A40), Ada (RTX 40xx, RTX 6000 Ada), datacenter Blackwell (B100, B200, GB100, GB200), or workstation Blackwell (RTX PRO 6000 Blackwell, RTX 5090), the default build will not contain native code for your card. The binary may still load via PTX JIT for some architectures, but for production use pass the matching sm_* value via -DCMAKE_CUDA_ARCHITECTURES=....

sm_121 requires a recent CUDA Toolkit

sm_121 (GB10 / DGX Spark) is only known to CUDA Toolkit 13.0 and newer. On CUDA 12.x the automatic append is suppressed, but if you explicitly pass -DCMAKE_CUDA_ARCHITECTURES="...;121" on an older toolkit you will get nvcc fatal: Unsupported gpu architecture 'compute_121'. Drop the 121 entry or upgrade the toolkit.

Step 5: Build, install, and verify

5.1 Build

cmake --build build -j"$(nproc)"

A clean build on a modern workstation takes a few minutes. The slowest single translation unit is src/kernels.cu.

5.2 Install

Install to a prefix of your choice. /opt/daqiri matches both the container and the Getting Started recipe:

sudo cmake --install build --prefix /opt/daqiri

Confirm the expected artifacts are in place:

ls /opt/daqiri/include/daqiri/daqiri.h
ls /opt/daqiri/lib/libdaqiri.so
ls /opt/daqiri/lib/cmake/daqiri/
ls /opt/daqiri/lib/pkgconfig/daqiri.pc
ldd /opt/daqiri/lib/libdaqiri.so | head

ldd should show librte_* (DPDK), libibverbs.so, librdmacm.so, libcudart.so, and libyaml-cpp.so resolving. Unresolved entries usually mean either DPDK was installed to a prefix not on the dynamic loader path (add it to /etc/ld.so.conf.d/ and re-run sudo ldconfig), or the RDMA libraries from Step 2 were not actually installed.

5.3 Smoke test

Verify the build with the standard two-port TX/RX loopback. This requires a NIC with two ports connected to each other by a physical SFP cable, and that you replace the <angle-bracket> placeholders in the YAML (PCIe BDFs, CPU cores, destination MAC) for your system. The walkthrough for those edits lives in Raw Ethernet Benchmarking → Update the loopback configuration; do that first, then run:

sudo ./build/examples/daqiri_bench_raw_gpudirect \
    ./build/examples/daqiri_bench_raw_tx_rx.yaml \
    --seconds 5

A successful run prints a stream of [INFO] lines followed by an RX/TX rate summary with non-zero packet counts on both sides. If the program aborts immediately with EAL: No free hugepages reported, see Step 6: Troubleshooting below.

DGX Spark

On DGX Spark, use daqiri_bench_raw_tx_rx.yaml to understand the complete TX/RX configuration. Adapt the NIC addresses, use host-pinned memory, select the Spark cores, and set the destination MAC to the RX port. Generate a raw-Ethernet pair can apply those parameters for you.

No NIC available?

If you don't have a NIC at all and just want to verify the binary starts cleanly, you can substitute daqiri_bench_raw_sw_loopback.yaml. It runs an in-process TX/RX through DPDK's software loopback with no hardware required. It is not representative of production performance and should not replace the real loopback above for build acceptance.

Step 6: Troubleshooting

CMake: Could not find a package configuration file provided by ... libdpdk

pkg-config cannot locate the DPDK installation. Either point it at the prefix you used in Step 3.3:

export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:/usr/local/lib/x86_64-linux-gnu/pkgconfig

or, if you installed DPDK from MLNX_OFED to /opt/mellanox/dpdk, rely on the fallback in src/CMakeLists.txt (search for mellanox); no extra environment variable is needed, but the path must exist.

CMake: Could not find libibverbs / librdmacm

The RDMA development headers are missing. Re-run the apt install from Step 2:

sudo apt install -y libibverbs-dev librdmacm-dev libmlx5-1

If apt reports the packages are "installed" but the -dev symlinks are missing on disk (this happens on some base images such as nvcr.io/nvidia/pytorch), force a reinstall from the DOCA repository configured in Step 1:

sudo apt install --reinstall libibverbs-dev librdmacm-dev libmlx5-1

nvcc fatal: Unsupported gpu architecture 'compute_121'

Your CUDA Toolkit predates GB10 / Blackwell support but sm_121 ended up in the architecture list. Either omit 121 from CMAKE_CUDA_ARCHITECTURES or pin an explicit list that matches your GPU:

cmake -S . -B build ... -DCMAKE_CUDA_ARCHITECTURES="80;90"

The default in src/CMakeLists.txt already suppresses the 121 auto-append on CUDA < 13.0, so this error generally only appears when the override was provided explicitly. See the GPU compute capability table for valid values.

Runtime: EAL: Cannot get hugepage information or EAL: No free hugepages reported

DPDK could not allocate hugepages because none are reserved (Cannot get hugepage information when nr_hugepages is zero, or No free hugepages reported when the reserved pool is exhausted). Hugepage configuration is part of system setup, not the build. Follow System Configuration → Step 4: Enable Huge pages to add a persistent reservation, or reserve some temporarily with:

echo 1024 | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

Runtime: nvidia-fs not loaded (with DAQIRI_ENABLE_GDS=ON)

The cuFile path needs the nvidia-fs kernel module loaded and a GDS-supported destination filesystem. Verify with:

lsmod | grep nvidia_fs
/usr/local/cuda/gds/tools/gdscheck.py -p

Full setup notes are in DAQIRI_ENABLE_GDS: cuFile burst writes. If nvidia-fs cannot be loaded on this host, drop -DDAQIRI_ENABLE_GDS=ON from the CMake configure step. Host-memory burst writes do not require it.

pkg-config: command not found

sudo apt install -y pkgconf

DPDK meson setup: ERROR: Dependency \"libnuma\" not found

The Dockerfile installs libnuma-dev in the same apt batch as the DPDK build tooling. If you skipped it:

sudo apt install -y libnuma-dev

Build succeeds but ibv_devinfo reports 0 HCAs found

The user-space libraries are installed but the kernel drivers are not; lsmod | grep mlx5 will be empty. Follow Check your NIC drivers to install doca-ofed and reboot.

Platform-specific notes

The build recipe above is the same on every supported host. The notes below cover the few places where defaults or expectations diverge. Open the tab that matches your hardware.

DGX Spark (GB10)IGX Orin + dGPUx86_64 RTX Pro Server

The integrated ConnectX-7 appears in ibv_devinfo as one or two mlx5_* HCAs depending on link configuration. No separate driver install beyond the DOCA repository setup is needed.
GB10 is compute capability 12.1 (sm_121). DAQIRI's default arch list adds 121 automatically when configuring with CUDA Toolkit 13.0 or newer; on those toolkits no override is needed. On older toolkits, GB10 is not supported.
DGX Spark uses NVLink-C2C unified memory and has no separate GPU BAR1, so generate data buffers with --memory-kind host_pinned rather than device.
nvidia-peermem is not used; GPUDirect goes through the dma-buf path enabled by the DPDK patches in Step 3.
For a runnable end-to-end test after the build completes, start with daqiri_bench_raw_tx_rx.yaml or daqiri_bench_rdma_tx_rx.yaml, then follow the DGX Spark profile callout to adapt or generate a concrete pair from the discovered system parameters.

The reference dGPU is the RTX 6000 Ada (compute capability 8.9), which is not in the default arch list. Pass -DCMAKE_CUDA_ARCHITECTURES=89 on the CMake configure step:
```
cmake -S . -B build ... -DCMAKE_CUDA_ARCHITECTURES=89
```
Use a different value (or extra entries) if you have a different dGPU installed.
On IGX Orin, several system settings (BAR1 size, MRRS, NIC link layer) need to be applied before DAQIRI will run end-to-end. Follow the IGX Series tab in System Configuration for the full sequence. The build itself does not require those changes.
GPUDirect can use either the patched DPDK's dma-buf path (recommended) or the legacy nvidia-peermem module. The patched DPDK built in Step 3 removes the peermem dependency.

For ARM64 (aarch64) hosts, set PKG_CONFIG_PATH to the aarch64 directory:

export PKG_CONFIG_PATH=/usr/local/lib/aarch64-linux-gnu/pkgconfig:${PKG_CONFIG_PATH:-}

The IGX/Tegra kernel does not expose /proc/config.gz; use the /boot/config-$(uname -r) fallback shown in Prerequisite verification to confirm CONFIG_DMA_SHARED_BUFFER=y.

"RTX Pro Server" covers any x86_64 workstation or server with a ConnectX-6 Dx (or later) NIC and an RTX Pro / Workstation GPU. Confirm nvidia-smi reports a GPUDirect-capable GPU (any RTX Pro / Quadro / Data Center class card, not GeForce, and see the warning in Concepts → GPUDirect).
Set -DCMAKE_CUDA_ARCHITECTURES to match the installed card. RTX PRO 6000 Blackwell Server/Workstation Edition is 120 (workstation Blackwell, sm_120); RTX 6000 Ada is 89; RTX A6000 is 86. Confirm with nvidia-smi --query-gpu=compute_cap --format=csv,noheader (e.g. 12.0 → 120):
```
cmake -S . -B build ... -DCMAKE_CUDA_ARCHITECTURES=120
```
x86_64 hosts use /usr/local/lib/x86_64-linux-gnu/pkgconfig for the DPDK .pc file. That's the default PKG_CONFIG_PATH entry shown in Step 3.4.
All other steps are identical to the generic recipe above.

Next steps

Once libdaqiri.so is installed and the smoke test passes:

System Configuration: tune the host (hugepages, NIC link layer, GPU BAR1, CPU isolation) for production performance.
Raw Ethernet Benchmarking: run daqiri_bench_raw_gpudirect over a physical loopback.
Understanding the Configuration File: pick the right starter YAML for your use case from the decision tree.

Cleanup

To remove the CMake install while keeping every prerequisite (DPDK, DOCA libraries, CUDA, hugepages, NIC drivers) in place, run scripts/cleanup.sh with the cmake target:

scripts/cleanup.sh cmake             # interactive, with manifest preview
scripts/cleanup.sh cmake --dry-run   # show what would be removed
scripts/cleanup.sh cmake --yes       # non-interactive

The script reads build/install_manifest.txt (written by Step 5.2) for the canonical list of installed paths, then refuses to remove anything unless every manifest entry is under DAQIRI_PREFIX. Manifest entries that are already absent are reported and skipped so cleanup can be rerun after a partial removal. Verification still runs and decides the final exit status. Override the install prefix with DAQIRI_PREFIX=... if you installed somewhere other than /opt/daqiri, or BUILD_DIR=... if your build tree is named something other than build. When the manifest is missing, the script falls back to a name-scoped scan that auto-removes only DAQIRI-owned artifacts and flags vendored spdlog/, yaml-cpp/, and libyaml-cpp.so* for manual review. The final step runs verification (ls /opt/daqiri, pkg-config --modversion daqiri, ldconfig -p | grep daqiri) and exits non-zero if any DAQIRI artifact is still present.

To remove only DAQIRI's container image, use the container target:

scripts/cleanup.sh container             # interactive
scripts/cleanup.sh container --dry-run   # show what would be removed

Override IMAGE_TAG=... if you built with a non-default tag. Pass all instead of cmake or container to remove both the CMake install and the container image in one run.

DAQIRI overview