--- name: rtvi-cv-customize-model description: How to swap the DeepStream CV detection model in the VSS Alerts Blueprint verification (2d_cv) mode - covers ONNX export, custom bbox parsers, compose mount gotchas, nvinfer config, runtime TRT engine build, deployment, and a segmentation-capable model addendum handoff. metadata: team: accelerated-microservices author: "NVIDIA CORPORATION " tags: - vss - cv-customization - rtvi-cv-customize-model - alerts-blueprint languages: - en domain: computer-vision --- # CV Detection Model Customization — VSS Alerts Blueprint (`2d_cv` mode only) The RT-CV perception container (`vss-rt-cv`) runs a DeepStream pipeline with a configurable primary GIE (GPU Inference Engine). By default it uses GDINO or RTDETR. This guide covers replacing it with any ONNX-format model, using YOLOv11 COCO 80 as the worked example. This only applies to `--mode verification` (`2d_cv`). Real-time alerts mode (`2d_vlm`) has no CV detector. For instance-segmentation or detection-plus-mask models, complete [references/segmentation-model-contract.md](references/segmentation-model-contract.md) before writing parser or handoff code. --- ## When to use Use this skill when the user wants to: - swap the stock `vss-rt-cv` detector for another ONNX model in verification mode, - debug a broken ONNX staging path, ghost-directory bind mount, or missing runtime TRT engine build, - fix a DeepStream parser load failure such as `dlsym failed` on the bbox parser symbol. Do **not** use this skill for: - `2d_vlm` real-time alerts mode, - scaffolding a brand-new standalone RTVI CV microservice (use `rtvi-cv-scaffold-vss-service`). ## Instructions - Keep the answer scoped to VSS Alerts Blueprint verification mode (`2d_cv`) unless the user explicitly asks to compare modes. - Paths beginning with `deploy/docker/` are relative to the [VSS Blueprint repository](https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization), not the DeepStream repository. Clone or reuse a VSS checkout that is **v3.2.1 or compatible**, then run these commands from that repository root (see [VSS Quickstart](https://docs.nvidia.com/vss/latest/quickstart.html#download-the-deployment-package)). - If the user asks about ONNX staging or a compose mount, explicitly say that mounting a missing file path is wrong: use the stock parent-directory mount, keep the ONNX under `${VSS_DATA_DIR}/models/yolo`, and re-stage the ONNX file after any `dev-profile.sh up` that recreates the models directory. - If the user asks about parser load failures, line up `parse-bbox-func-name`, the `extern "C"` function symbol, and `CHECK_CUSTOM_PARSE_FUNC_PROTOTYPE(...)`, then rebuild the `.so` with both DeepStream and CUDA include paths present. - If the model is detection-plus-mask or instance segmentation, route the mask-contract decision first through [references/segmentation-model-contract.md](references/segmentation-model-contract.md) before editing parser or handoff logic. ## Examples - "Replace the VSS Alerts Blueprint verification detector with a YOLO11 ONNX model and redeploy `perception-alerts`." - "DeepStream says `dlsym failed` for the bbox parser after loading the custom parser library. What should I check?" - "I mounted the ONNX file path directly in compose before the host file existed. Is that okay?" --- ## Architecture ``` NVStreamer (RTSP) → SDR (port 9010) → DeepStream (perception-alerts) └─ primary-gie (nvinfer) ├─ ONNX → TRT engine (built once, then cached) ├─ custom bbox parser (.so) └─ label file (.txt) └─ Kafka → mdx-raw └─ vss-behavior-analytics └─ mdx-incidents └─ vlm-as-verifier ``` ## VSS source location This skill is documentation-only and does not ship the VSS deployment sources. Use a VSS **v3.2.1 or compatible** checkout; run all paths and commands from that repository root. Clone/LFS setup, stock Alerts profile files, and the YOLOv11 customization file tree are in [references/vss-source-layout.md](references/vss-source-layout.md). --- ## Adapting to a Different Model YOLOv11 COCO 80 is the worked example throughout all steps below. The same pattern applies to any ONNX-format detector — substitute at these points: | Step | What to change | |------|----------------| | **Step 1** | Replace the export procedure with whatever your model's training library requires. Confirm the resulting `.onnx` exists on the host before continuing. | | **Steps 2–3** | Inspect your model's actual output tensor name, shape, layout, and whether NMS is applied in-graph. Do not assume it matches YOLOv11. The `deepstream-dev` skill ([`skills/deepstream-dev/`](../deepstream-dev/)) has a generation-by-generation YOLO output format table and `references/nvinfer_config.md` for full property reference. | | **Step 4** | Update `output-blob-names` to your tensor name, `infer-dims` to your input shape, and `cluster-mode` to match whether NMS is in-graph (`4`) or not (`2`). | | **Step 5** | Update the `--onnx` path, the input tensor name in `--minShapes`/`--optShapes`/`--maxShapes` (not `images` for non-YOLO models), and the `sed` commands to reference your nvinfer `.txt` file. Add a new `if [[ $MODEL_NAME_2D == "YOURMODEL" ]]` block rather than editing the YOLO block. | --- ## Step 1: Stage Your ONNX Model (Host, One-Time) Obtain a TensorRT-compatible ONNX model and place it at `${VSS_DATA_DIR}/models/yolo/yolo11s.onnx` before continuing. See [references/yolov11-onnx-export.md](references/yolov11-onnx-export.md) for the export settings and tensor layout used in this reference. Adapt the parser and nvinfer config in later steps to match your model's actual output. After every full `dev-profile.sh up`, restore ownership of the recreated `models/yolo` directory and re-stage your ONNX file: ```bash sudo mkdir -p "${VSS_DATA_DIR}/models/yolo" sudo chown -R $(id -u):$(id -g) "${VSS_DATA_DIR}/models/yolo" # Copy your ONNX model to ${VSS_DATA_DIR}/models/yolo/yolo11s.onnx ``` --- ## Steps 2–3: Inspect the Model Output and Build the Parser Read and follow [references/yolov11-parser.md](references/yolov11-parser.md) before creating `${VSS_PROFILE_DIR}/deepstream/custom_parser/nvdsparseyolov11.cpp` or editing the parser build in `Dockerfiles/perception.Dockerfile`. Before writing any parser code, confirm these four things for your model: - **Output tensor name and shape** — drives `output-blob-names` and `infer-dims` in Step 4 - **Pre-NMS or post-NMS** — drives `cluster-mode` in Step 4; see `deepstream-dev` Rule 13 for the generation-by-generation breakdown - **Coordinate semantics** — `cx/cy/w/h` center-format or `x1/y1/x2/y2` corner-format; the YOLOv11 example uses center-format (see [references/yolov11-parser.md](references/yolov11-parser.md)) - **Parser symbol name** — the `extern "C"` function name must match `parse-bbox-func-name` in Step 4 and `CHECK_CUSTOM_PARSE_FUNC_PROTOTYPE(...)` exactly Confirm the model output tensor layout instead of assuming it matches YOLOv11. Keep the parser symbol identical in the exported C++ function, `CHECK_CUSTOM_PARSE_FUNC_PROTOTYPE(...)`, and the Step 4 `parse-bbox-func-name` setting. For instance segmentation or detection-plus-mask models, complete the contract decisions table in [references/segmentation-model-contract.md](references/segmentation-model-contract.md) before writing parser or handoff code. --- ## Step 4: nvinfer Config `${VSS_PROFILE_DIR}/deepstream/configs/yolov11.txt` — the primary GIE sub-config referenced by the DeepStream run config. **`model-engine-file` and `batch-size` are patched at container startup by the `ds-start.sh` block in Step 5.** Copy the values below as-is; Step 5 will overwrite both with the correct engine path and sensor count before DeepStream reads the file. ```ini [property] gpu-id=0 net-scale-factor=0.0039215697906911373 # 1/255 input normalization model-engine-file=/opt/storage/yolo11s_fp16.engine onnx-file=/opt/storage/yolo/yolo11s.onnx batch-size=1 # patched at startup from NUM_SENSORS network-mode=2 # 0=FP32, 1=INT8, 2=FP16 network-type=0 # 0=Detector num-detected-classes=80 interval=0 gie-unique-id=1 output-blob-names=output0 # must match your model's output tensor name infer-dims=3;640;640 # C;H;W maintain-aspect-ratio=1 parse-bbox-func-name=NvDsInferParseCustomYoloE # must match extern "C" function name in .so custom-lib-path=/opt/deepstream-yolo/libnvdsparseyolov11.so labelfile-path=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/metropolis_perception_app/mounted-configs/yolo-coco-labels.txt cluster-mode=2 # 2=NMS — required: Ultralytics ONNX has no NMS in-graph [class-attrs-all] pre-cluster-threshold=0.25 topk=300 nms-iou-threshold=0.45 ``` --- ## Step 5: Runtime TRT Engine Build (`ds-start.sh`) TRT engines are GPU-architecture-specific. Build the engine on first startup when it is missing, then reuse it on subsequent restarts. Rebuild an existing engine only when `FORCE_REBUILD=true`. The stock Alerts compose file bind-mounts `${VSS_DATA_DIR}/models/` at `/opt/storage/`, so the generated engine persists on the host. **Which `ds-start.sh`?** Stock Alerts runs `services/rtvi/rtvi-cv/ds-start.sh` (via `extends`), not `${VSS_PROFILE_DIR}/deepstream/init-scripts/ds-start.sh`. This skill edits the profile copy, then remounts it in Step 6 so those edits actually run. Full map of the three paths, remount trade-offs, and `DS_CONFIG_FILE` pairing: [references/ds-start-entrypoint.md](references/ds-start-entrypoint.md). Edit `${VSS_PROFILE_DIR}/deepstream/init-scripts/ds-start.sh`, then override the stock rtvi-cv `ds-start.sh` bind mount in the Alerts profile compose file as shown in Step 6. Without that override, profile-script edits never execute. **Critical — honor `DS_CONFIG_FILE`.** The profile script sets `CONFIG_FILE=${1:-...}` and never reads `DS_CONFIG_FILE`. The Alerts compose `command` invokes `ds-start.sh` with no positional args, while Step 6 sets `DS_CONFIG_FILE` to the absolute **mounted-configs** run config (not the stock `.../configs/...` path). Replace the script's existing `CONFIG_FILE=...` assignment with: ```bash # Prefer $1 when provided; otherwise use DS_CONFIG_FILE from compose (absolute path). # Without this, YOLO_CONFIG resolves to ./yolov11.txt and the stock detector keeps running. CONFIG_FILE="${1:-${DS_CONFIG_FILE:-/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/metropolis_perception_app/mounted-configs/run_config-api-rtdetr-protobuf.txt}}" ``` Then add the YOLO block (before the GDINO/RT-DETR model branches is fine): ```bash if [[ $MODEL_NAME_2D == "YOLO" ]]; then YOLO_ONNX=/opt/storage/yolo/yolo11s.onnx YOLO_ENGINE=/opt/storage/yolo11s_fp16.engine YOLO_ENGINE_TMP="${YOLO_ENGINE}.building" FORCE_REBUILD=${FORCE_REBUILD:-false} if [[ ! -f "$YOLO_ONNX" ]]; then echo "ERROR: ONNX not found at ${YOLO_ONNX}. Stage your model there before starting." exit 1 fi if [[ ! -s "$YOLO_ENGINE" || "${FORCE_REBUILD,,}" == "true" ]]; then echo "Building TensorRT engine: ${YOLO_ENGINE}" rm -f "$YOLO_ENGINE_TMP" if /usr/src/tensorrt/bin/trtexec \ --onnx=${YOLO_ONNX} \ --minShapes=images:1x3x640x640 \ --optShapes=images:${NUM_SENSORS}x3x640x640 \ --maxShapes=images:${NUM_SENSORS}x3x640x640 \ --fp16 --saveEngine=${YOLO_ENGINE_TMP}; then mv "$YOLO_ENGINE_TMP" "$YOLO_ENGINE" else rm -f "$YOLO_ENGINE_TMP" echo "ERROR: TensorRT engine build failed; existing engine was preserved." exit 1 fi else echo "Reusing cached TensorRT engine: ${YOLO_ENGINE}" fi # Patch the run config to use yolov11.txt for the primary GIE # (relative name is intentional: DeepStream resolves it against CONFIG_FILE's dir) sed -i '/^\[primary-gie\]/,/^\[/{s/config-file=.*/config-file=yolov11.txt/;}' "$CONFIG_FILE" # Patch engine path and batch size into yolov11.txt (same dir as CONFIG_FILE) YOLO_CONFIG="$(dirname "${CONFIG_FILE}")/yolov11.txt" sed -i "s|model-engine-file=.*|model-engine-file=${YOLO_ENGINE}|" "${YOLO_CONFIG}" sed -i "/^\[property\]/,/^\[/{s/^batch-size=.*/batch-size=${NUM_SENSORS}/;}" "${YOLO_CONFIG}" fi ``` To add a different model, add a new `if [[ $MODEL_NAME_2D == "YOURMODEL" ]]` block. Update: - `--onnx` path and `--saveEngine` output path - `--minShapes`/`--optShapes`/`--maxShapes` input tensor name and dimensions for your model - The `sed` commands to point at your nvinfer `.txt` file --- ## Step 6: Compose + Env In the existing `${VSS_PROFILE_DIR}/compose.yml`, update the `perception-alerts` service: ```yaml perception-alerts: build: context: $VSS_APPS_DIR/developer-profiles/dev-profile-alerts dockerfile: Dockerfiles/perception.Dockerfile volumes: # Keep the stock parent-directory mount. Do not replace it with a direct # mount of yolo11s.onnx: Docker creates a ghost directory if the host # file does not exist. - $VSS_DATA_DIR/models/:/opt/storage/ - $VSS_APPS_DIR/developer-profiles/dev-profile-alerts/deepstream/configs/:/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/metropolis_perception_app/mounted-configs/ # Override stock rtvi-cv ds-start.sh (extends bind). Without this, # edits under deepstream/init-scripts/ never run — see # references/ds-start-entrypoint.md. - $VSS_APPS_DIR/developer-profiles/dev-profile-alerts/deepstream/init-scripts/ds-start.sh:/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/metropolis_perception_app/ds-start.sh:ro environment: MODEL_NAME_2D: ${MODEL_NAME_2D} NUM_SENSORS: ${NUM_SENSORS} FORCE_REBUILD: ${FORCE_REBUILD:-false} # Use mounted-configs (where yolov11.txt lives), not stock .../configs/... DS_CONFIG_FILE: /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/metropolis_perception_app/mounted-configs/run_config-api-rtdetr-protobuf.txt ``` Set in `${VSS_PROFILE_DIR}/.env`: ```bash MODEL_NAME_2D="YOLO" NUM_SENSORS=1 FORCE_REBUILD=false ``` Keep `FORCE_REBUILD=false` for normal starts. Compose's `--force-recreate` option recreates the container but does **not** rebuild the TensorRT engine unless `FORCE_REBUILD=true`. **Gotcha — `generated.env` wins at restart.** Step 7's redeploy uses `--env-file .../generated.env`, which is produced by `dev-profile.sh` from `.env`. Editing `.env` alone does **not** change `FORCE_REBUILD` for that compose command. For a one-shot engine rebuild, pass the override on the shell (see Step 7). To persist the value for later `dev-profile.sh` runs, edit `.env` and re-run `dev-profile.sh` so it regenerates `generated.env`. --- ## Step 7: Deploy Run from the video-search-and-summarization repository root (`VSS_ROOT` / `VSS_DEPLOY_DIR` / `VSS_PROFILE_DIR` from VSS source location): ```bash : "${VSS_ROOT:=$PWD}" : "${VSS_DEPLOY_DIR:=${VSS_ROOT}/deploy/docker}" : "${VSS_PROFILE_DIR:=${VSS_DEPLOY_DIR}/developer-profiles/dev-profile-alerts}" ``` **1. Single-GPU host (only GPU 0):** edit `${VSS_PROFILE_DIR}/.env` so these keys match the required final values below. Edit the source profile env file, not `generated.env` (`dev-profile.sh` regenerates that). Stock Alerts reserves GPU 0 and puts RT-VLM / LLM / VLM on GPU 1; `dev-profile.sh` CLI flags cannot clear `RESERVED_DEVICE_IDS` or set `FIXED_SHARED_DEVICE_IDS`. Confirm device IDs exist with `nvidia-smi --query-gpu=index --format=csv,noheader,nounits` before deploying. ```text RESERVED_DEVICE_IDS='' FIXED_SHARED_DEVICE_IDS='0' RT_CV_DEVICE_ID='0' RT_VLM_DEVICE_ID='0' LLM_DEVICE_ID='0' VLM_DEVICE_ID='0' ``` **2. Deploy** the official Alerts verification profile. Select the hardware profile supported by your host, as documented by the VSS Quickstart. Do not present the remaining commands as a copy-paste handoff unless execution is blocked. ```bash "${VSS_DEPLOY_DIR}/scripts/dev-profile.sh" up \ -p alerts \ -m verification \ -H # 3. dev-profile.sh recreates the model directory. Restore ownership and # re-stage your ONNX model. export VSS_DATA_DIR="${VSS_DEPLOY_DIR}/data-dir" sudo mkdir -p "${VSS_DATA_DIR}/models/yolo" sudo chown -R $(id -u):$(id -g) "${VSS_DATA_DIR}/models/yolo" # Copy your ONNX model to ${VSS_DATA_DIR}/models/yolo/yolo11s.onnx # 4. Rebuild and recreate the customized perception service using the compose # project generated by dev-profile.sh. cd "${VSS_DEPLOY_DIR}" docker compose \ --env-file developer-profiles/dev-profile-alerts/generated.env \ up -d --build --force-recreate perception-alerts # One-shot TensorRT rebuild (after ONNX / shape / GPU / TRT changes). # Inline FORCE_REBUILD overrides generated.env for this invocation only: # FORCE_REBUILD=true docker compose \ # --env-file developer-profiles/dev-profile-alerts/generated.env \ # up -d --build --force-recreate perception-alerts ``` TRT engine build takes approximately 15–30 seconds on Blackwell hardware. Subsequent restarts reuse the engine persisted under `${VSS_DATA_DIR}/models/`. --- ## Common Gotchas See [references/common-gotchas.md](references/common-gotchas.md) for single-GPU reservation failures, wipe/restage after `dev-profile.sh up`, ghost file mounts, `dlsym` parser mismatches, and Redis stream contamination.