Legacy example · C++ · Python · Version 1.0
UCX-based Distributed Endoscopy Tool Tracking
| Authors | Holoscan Team (NVIDIA) |
|---|---|
| Supported platforms | x86_64, aarch64 |
| Language | C++, Python |
| Version | 1.0 |
| Compatibility | HSDK 4.6 |
| Contribution level | Level 1 · Very high-quality code |
| Source | HoloHub |
Legacy HoloHub application
This application comes from HoloHub and was last tested with Holoscan SDK (HSDK) 4.6. It runs with the HoloHub CLI, ./holohub.
Digital endoscopy is a key technology for medical screenings and minimally invasive surgeries. Using real-time AI workflows to process and analyze the video signal produced by the endoscopic camera, this technology helps medical professionals with anomaly detection and measurements, image enhancements, alerts, and analytics.
Fig. 1 Endoscopy (laparoscopy) image from a cholecystectomy (gallbladder removal surgery) showing AI-powered frame-by-frame tool identification and tracking. Image courtesy of Research Group Camma, IHU Strasbourg and the University of Strasbourg (NGC Resource)
The Distributed Endoscopy Tool Tracking application provides an example of how an endoscopy data stream can be captured and processed using the C++ or Python APIs on multiple hardware platforms.
The Distributed Endoscopy Tool Tracking application is very similar to the Endoscopy Tool Tracking application but divides all operators into three fragments as depicted below. This allows fragments to run on different systems. For example, one can run the Video Input Fragment and the Visualization Fragment on a radiology workstation while the Inference Fragment runs on a more powerful system with multiple GPUs. Refer to the Holoscan SDK User Guide for more information on distributed applications.
Video Stream Replayer Input
Fig. 2 Tool tracking application workflow with replay from file
The pipeline uses a recorded endoscopy video file (generated by convert_video_to_gxf_entities script) for input frames. Each input frame in the file is loaded by Video Stream Replayer and passed to the following two pipeline branches:
- In the first branch, the input frames are directly passed to Holoviz in the Visualization Fragment for rendering in the background.
- In the second branch, the frames go through the Format Converter in the Inference Fragment to convert the data type of the image from
uint8tofloat32before it is fed to the tool tracking model (with Custom TensorRT Inference). The result is then ingested by the Tool Tracking Postprocessor which extracts the masks, points, and text from the inference output, before Holoviz renders them as overlays in the Visualization Fragment.