Endoscopy 3D Depth Visualization

Interactive 3D depth-surface visualizations of endoscopy images and video, viewable
in any browser. Two scripts, each producing a self-contained Plotly HTML file you
can rotate, zoom, and (for video) animate:

• depth_to_3d.py — single image → rotatable 3D depth surface
• depth_3d_video.py — endoscopy video → animated 3D depth surface that
  updates as the scope moves

Both include automatic black-border / text removal (the circular endoscope field
of view and overlaid annotations) so only tissue is rendered.

---

Quick start

1. Install dependencies

pip install -r requirements.txt

PyTorch: The requirements.txt pins are generic. For GPU support, install
the CUDA-matched build from pytorch.org first:

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124

2. Clone the Depth-Anything-V2 package

The single-image script uses the original Depth-Anything-V2
PyTorch implementation (it produces finer-grained depth than the HF transformers
port). Clone it next to the scripts:

git clone https://github.com/DepthAnything/Depth-Anything-V2.git depth_anything_v2_repo

Model weights (~1.3 GB for the Large encoder) are downloaded automatically from
HuggingFace on first run and cached locally.

3. Run

Single image:

python depth_to_3d.py --image my_endoscopy.jpg --output result.html

Video (illumination-based depth — no model, instant):

python depth_3d_video.py --video my_endoscopy.mp4 --output result.html

Open the resulting .html file in a browser.

---

How it works

Single image (depth_to_3d.py)

1. Detects the tissue region (largest connected non-black blob — excludes black
   corners and overlaid text automatically; no-op if there’s no border).
2. Crops to the tissue bounding box and runs Depth-Anything-V2 (Large) for
   relative depth.
3. Builds a Plotly Surface where X = image column, Y = image row, Z = depth,
   colored by depth with the Spectral_r colormap (matching the official demo).
4. Non-tissue cells are set to NaN so only tissue renders.

Depth-Anything-V2 outputs relative (affine-invariant) depth, not metric — good
for shape visualization, not measurement.

Video (depth_3d_video.py)

Samples frames from the video at a fixed interval and builds an animated 3D
surface that morphs as the scope moves (Play / Pause / Restart buttons + a
timeline slider). Two depth methods:

--method brightness (default — recommended for endoscopy)

Illumination-based depth. The endoscope’s light source is co-located with the
camera, so by the inverse-square law brightness falls off with distance: dark
lumen (tunnel opening) = far = deep depression; bright near-tissue (wall) = close
= raised surface. This is physically motivated for endoscopy and naturally
produces a tube shape with visible tunnel openings. No model or GPU needed.

Tuning knobs for tunnel depth and spike removal:

Flag	Default	Effect
--gamma	0.5	Brightness→depth curve. 0.5 = √ (physical inverse-square). Higher (1.0–2.0) = deeper tunnel depressions.
--clahe-clip	3.0	Local contrast enhancement (CLAHE). The main lever for tunnel depth — stretches contrast locally so dark tunnels stand out. 0 = off.
--shadow-gain	1.5	Pushes tunnel floors deeper without affecting bright walls. 1.0 = off. Higher (2–4) = more dramatic.
--despike	2	Morphological opening radius — removes bubble/reflection spikes on the grid. 0 = off.
--specular-radius	5	Top-hat radius for detecting bubble reflections before CLAHE. 0 = off.
--blur	3.0	Spatial Gaussian blur sigma.
--erode	10	Pixels to erode the tissue mask inward (cuts border drop-off).

--method depth-anything (opt-in — not recommended)

Uses Depth-Anything-V2 per frame. Not recommended for endoscopy — it was
trained on ordinary photos and produces a radial bowl artifact (center-far,
edge-close plateau) with little meaningful local detail. Included for comparison.

---

Examples

Single image

# Basic
python depth_to_3d.py --image endoscopy.jpg

# Exaggerate depth relief
python depth_to_3d.py --image endoscopy.jpg --z-scale 2.0

# Disable border removal
python depth_to_3d.py --image endoscopy.jpg --no-crop-border

# Smaller/faster model
python depth_to_3d.py --image endoscopy.jpg --encoder vitb

# Self-contained HTML (no internet needed to view)
python depth_to_3d.py --image endoscopy.jpg --offline

Video

# Default (illumination depth, moderate tunnel relief)
python depth_3d_video.py --video endoscopy.mp4

# Deeper tunnels + spike removal
python depth_3d_video.py --video endoscopy.mp4 
    --gamma 1.0 --clahe-clip 4.0 --shadow-gain 2.5 --despike 3

# Compare with Depth-Anything-V2 (needs GPU)
python depth_3d_video.py --video endoscopy.mp4 --method depth-anything --half

# Sub-segment + denser frames
python depth_3d_video.py --video endoscopy.mp4 
    --start-sec 10 --end-sec 25 --sample-sec 0.5

# Flip Z (tunnels as peaks instead of dips)
python depth_3d_video.py --video endoscopy.mp4 --invert

# Self-contained HTML
python depth_3d_video.py --video endoscopy.mp4 --offline

---

Sample data

This repository does not include sample endoscopy images or video. Provide
your own:

• Image: any RGB endoscopy frame (.jpg, .png). Works with or without a
  black border.
• Video: any video file OpenCV can read (.mp4, .avi, …). The scope
  should move slowly through the lumen.

Privacy: Endoscopy recordings may contain identifiable patient information.
Ensure you have appropriate consent before using or sharing any medical data.

---

Requirements

Package	Purpose
opencv-python	Image/video I/O, tissue detection, morphology
numpy	Array math
Pillow	Image I/O (depth map export)
matplotlib	Spectral_r colormap
plotly	Interactive 3D surfaces + animation
huggingface-hub	Model weight download
torch, torchvision	Depth-Anything-V2 inference (single image + --method depth-anything)

A CUDA GPU is required for Depth-Anything-V2 (single-image script and
--method depth-anything). The illumination-based video method (--method brightness, the default) runs on CPU with no model.

Tested with Python 3.12, torch 2.6 (CUDA 12.4), on an NVIDIA RTX A5000.

---

Limitations

This is a proof of concept, not a clinical tool.

• Depth-Anything-V2 outputs relative (affine-invariant) depth, not metric.
  It was not trained on endoscopy and produces a radial bowl artifact on tube-like
  scenes. The illumination-based method is more anatomically sensible but is
  affected by tissue albedo (color) variations and is not metric either.
• No 3D reconstruction. The video script shows the current frame’s surface
  animated over time — it does not accumulate or stitch frames into a persistent
  model. Monocular endoscopy reconstruction requires metric depth, camera
  intrinsics, and motion estimation that are beyond this POC.
• Endoscope lens distortion (fisheye) is not corrected. Proper undistortion
  needs a calibration sequence (e.g., a chessboard recording through the scope).
• Bubble reflections can cause depth spikes. The --despike and
  --specular-radius flags mitigate this but cannot eliminate all artifacts.

License

MIT. The Depth-Anything-V2 package (cloned separately) is Apache 2.0.

---

Imported from gh:Kentucky-Open-Science/endoscopy-depth-prediction. Source last updated 2026-08-06. Synced 2026-08-06.