Lexington Digital Twin — Interactive 3D Viewer

Extract UE 4.24.3 UKy Campus LiDAR point cloud + DTM terrain tiles + aerial
imagery into open formats and view them in an interactive Three.js web viewer
— no Unreal Engine required at runtime. The twin now extends past campus to the full
Lexington / Lextran service area: authoritative LFUCG building footprints and street
centerlines, KyFromAbove/KYAPED-derived elevations, active LFUCG traffic-signal
locations, attributed OpenStreetMap crossing evidence, and the live Lextran bus feed.
Large city layers stay packed into one buffer / draw call per layer, with adaptive
quality controls protecting the viewer’s 30 fps interaction target.

Gallery

Google Photorealistic 3D Tiles streamed into the Three.js twin — the whole city in
real, textured 3D (campus, downtown skyline, the Lextran service area). This optional
layer stays off until enabled with a valid provider key; its cache is transient.

Oblique over campus toward the downtown skyline — real building textures, trees, and
terrain, with the live overlays (roads, signals, cameras, buses) draped on top.

Top-down: the generated road network, intersections and signals draped onto the
photorealistic surface at real elevation (so overpasses stay above the roads beneath).

The earlier OSM/LiDAR base map (still available): street level following a live Route 7
bus. Signal locations are anchored to LFUCG records; explicit crossing evidence comes
from attributed OSM data, while any generated markings and timings are labeled models.

Quick start

python -m tools.twin_server --detect-device auto
# Open http://localhost:8000/

tools/twin_server.py is one server that serves the static viewer, proxies the live
Lexington (Lextran) GTFS-Realtime feed so you get moving buses on the map, proxies the
city’s live traffic-camera stream URLs so you can watch real traffic at the
intersections (/api/cameras/*), and runs the authoritative shared-world agent API
(/api/world/*) — with solid geometry: agents collide with (and are pushed out of)
all buildings in the active pack via oriented per-building collision boxes, drones land
on roofs, and the ground model reports road / terrain / none per position. Add
--render for first-person agent cameras, --npc 12 for signal-aware background
traffic, --mock to
replay a recorded bus feed offline, --no-transit to skip the bus proxy, or
--no-cameras to skip the camera proxy. No buses or agents needed at all? Plain cd web && python -m http.server 8000 still works — routes, stops, and camera markers render; live buses and video just
stay off.

--detect-device auto selects CUDA FP16 when PyTorch can see an NVIDIA GPU and otherwise
falls back to CPU FP32. For an NVIDIA Container Toolkit installation, the equivalent
GPU-enabled container command is
docker compose -f docker-compose.yml -f docker-compose.gpu.yml up --build.

The viewer loads generated web/data/manifest.json and streams terrain tiles + LiDAR
point chunks. The runtime server serves active artifacts; it does not silently refresh
their public-data provenance.

What’s in here

CAMPUS/
├── LIDAR/                    # UE4 .uasset source (POINT_CLOUD_2019, 448 MB)
├── MESHES/DTM_GRID/          # UE4 .uasset sources (16 meshes, 18 textures, 17 materials)
├── tools/                    # Python extraction tools
│   ├── uasset.py             # Core UE4 package parser (v518 / 4.24.3)
│   ├── inspect.py            # Export/property dumper
│   ├── extract_texture.py    # Texture2D -> PNG -> JPEG
│   ├── extract_mesh.py       # StaticMesh -> .bin (positions, UVs, indices)
│   ├── extract_lidar.py      # LidarPointCloud -> decimated chunked .bin (campus only)
│   ├── ky_lidar.py           # KyFromAbove/KYAPED LiDAR -> citywide point cloud + ground grid
│   ├── extract_scene.py      # Blueprint scene assembly (transforms, materials)
│   ├── extract_buildings.py  # LiDAR building-class → 3D mesh (legacy, DBSCAN)
│   ├── extract_buildings_hybrid.py  # OSM footprints split LiDAR + give height (campus, UE source)
│   ├── build_city.py         # bootstrap height reference: OSM footprints + KYAPED heights
│   ├── public_data.py        # paginated LFUCG/KYTC downloads + provenance manifest
│   ├── lfucg_roads.py        # authoritative LFUCG centerlines → roads.json
│   ├── fetch_osm_crossings.py # attributed explicit OSM crossing evidence
│   ├── build_lfucg_buildings.py # LFUCG footprints + joined KyFromAbove heights → packed layer
│   ├── verify_buildings_osm.py      # verify footprints vs OSM ground truth
│   ├── osm_roads.py          # legacy OpenStreetMap campus-road fallback
│   ├── osm_city.py           # lightweight OSM city context + exact service bbox (city.json)
│   ├── lextran_gtfs.py       # Lextran static GTFS → routes + stops (transit.json)
│   ├── lex_cameras.py        # city traffic cams → intersection mapping (cameras.json)
│   ├── camera_detect.py      # batched YOLO26n → calibrated/road-gated twin traffic
│   ├── export_usd.py         # exact generated map layers → Omniverse-compatible USD
│   ├── twin_server.py        # one server: static viewer + shared-world API (/api/world/*)
│   │                         #   + live bus proxy (/api/transit/*) + camera URL proxy (/api/cameras/*)
│   ├── pack_buildings.py     # merge bootstrap building meshes → one buffer
│   ├── build_obb.py          # per-building oriented collision boxes (buildings.obb.json)
│   ├── roadgrid.py           # O(1) road-proximity index (on-road? nearest lane?) shared
│   │                         #   by the server, gym, and detector (mirrors web/roadgrid.js)
│   ├── transit_common.py     # shared lon/lat → scene projection (georef)
│   ├── roadnet.py            # road graph (routing) from roads.json
│   ├── traffic.py            # deterministic signals + NPC vehicles (sim core)
│   ├── verify_gym.py         # gym env test suite (check_env, vectorization, traffic, SPL, …)
│   ├── extract_roads.py      # aerial-texture road detector (alt. road source)
│   ├── fetch_lfucg_signals.py # legacy single-layer LFUCG signal downloader
│   ├── smooth_roads.py       # source-aware junction/crossing model → signals.json
│   ├── ground_buildings.py   # drop floating buildings onto the terrain (keep bridges)
│   ├── build_all.py          # Full pipeline orchestrator
│   ├── verify_viewer.py      # Headless viewer test (requires playwright)
│   └── verify_agents.py      # Headless agent-API sensor test (requires playwright)
├── web/                      # Three.js viewer (static)
│   ├── index.html
│   ├── app.js
│   ├── roads.js              # road ribbons + signals/crosswalks + live signal controller
│   ├── city.js               # city-wide OSM ground plane + streets (city.json)
│   ├── transit.js            # live Lextran buses + routes/stops/arrivals/alerts
│   ├── cameras.js            # traffic-camera markers + live HLS picture-in-picture
│   ├── agents.js             # autonomous agents (car/truck/robot/drone) + sensors (local)
│   ├── netagents.js          # renders the twin_server shared world (agents from any client)
│   ├── lorawan.js            # gateway planning, RF telemetry, and GeoJSON export
│   ├── rfcoverage.js         # deterministic placement + building-shaped RF geometry
│   ├── style.css
│   ├── lib/                  # Vendored Three.js 0.160 + OrbitControls + hls.js 1.6
│   └── data/                 # Generated extraction output
│       ├── manifest.json     # Unified scene manifest
│       ├── meshes/*.bin      # 16 terrain tiles (verts, UVs, indices)
│       ├── textures/*.jpg    # 18 aerial imagery textures
│       ├── lidar/chunk_*.bin # 64 decimated point-cloud chunks
│       ├── buildings/*.bin   # per-building meshes (legacy / fallback)
│       ├── buildings.pack.bin + .json  # active LFUCG-footprint building pack, one buffer
│       ├── buildings.obb.json # oriented collision box for each packed building
│       ├── roads.json        # authoritative LFUCG centerlines, smoothed + draped
│       ├── signals.json      # source-aware signal/crossing model (autonomous agents)
│       ├── city.json         # city-wide OSM streets + ground plane
│       ├── sources/          # downloaded public layers + hashes/provenance (generated)
│       ├── transit.json      # Lextran routes + stops (live buses via tools/twin_server.py)
│       └── cameras.json      # traffic cam → intersection map (live HLS via tools/twin_server.py)
├── campus_gym/              # Gymnasium (single) + PettingZoo (multi) env over the sim core
├── client/                  # twin.py — dependency-free Python client for the world API
├── examples/                # drone_demo.py + car/truck/robot vision demos (YOLO via the camera feed)
└── extracted/                # Per-domain manifests + reports

Re-extracting from source

Generated data lives under the gitignored web/data/. If it is absent or you need to
regenerate the campus georeference from the source assets:

pip install pillow numpy
python -m tools.build_all
# Or selectively:
python -m tools.build_all --skip-textures --skip-meshes

Filling in Lexington — KyFromAbove LiDAR

The campus point cloud started as a single UE asset (POINT_CLOUD_2019); the rest of
Lexington is filled in from the statewide KyFromAbove / KYAPED LiDAR (Phase 3),
which supersedes the campus scan where the sources overlap. tools/ky_lidar.py queries
the published tile index for the bbox in city.json, downloads 5,000-ft .copc.laz
tiles, reprojects them from NAD83 Kentucky Single Zone (US ft) into the scene’s UTM-16N
frame, and writes viewer point chunks plus the citywide ground.f32 elevation grid.
KyFromAbove data is distributed as public domain under its published
distribution policy;
the generated manifest records the source and transformation.

pip install "laspy[lazrs]" pyproj numpy pillow
python -m tools.ky_lidar --list                 # inspect tiles intersecting city.json
python -m tools.ky_lidar --download-aoi         # fetch every tile to extracted/ (large, resumable)
python -m tools.ky_lidar --build --heightmap    # -> web/data/lidar/ky_*.bin + ground.f32 + manifest
python -m tools.twin_server                     # serve viewer + world + buses on :8000

Then open http://localhost:8000/, expand the LiDAR panel, and tick visible. Agent
ground-snapping (tools/twin_server.Ground) uses the new ground.f32 grid as the primary
elevation citywide, so cars/buses follow real terrain beyond the campus tiles. Everything
lands under the gitignored web/data/; the previous campus manifest is backed up to
web/data/manifest.campus.bak.json.

Authoritative public-data rebuild

The public-data build requires an existing georeferenced web/data/manifest.json and
the extracted campus terrain; run tools.build_all first when rebuilding those from the
UE source assets. From that georeference onward, plan geometry and elevation come from
public GIS/LiDAR sources and every downstream layer is clipped to city.json.

Exact service bbox

The currently baked service extent is the exact city.json bbox_lonlat, ordered
[west, south, east, north]:

[-84.614899, 37.960404, -84.394988, 38.120465]

python -m tools.osm_city derives this from the current Lextran GTFS stops and shapes
(including its documented service margin), projects the matching scene bbox, and writes
the lightweight OSM street context. A later GTFS release can change the extent; builders
therefore read city.json rather than copying these coordinates into code. LFUCG roads
are clipped to that extent, LFUCG buildings are selected within it, and the OSM crossing
fetcher queries it without padding.

Source precedence and modeled fields

Domain	Primary evidence	Secondary/model behavior
Building plan geometry	LFUCG Building polygons	No OSM polygon replaces an LFUCG footprint. Heights are joined from the prior KyFromAbove-derived pack; nearby/default fallbacks are labeled per building.
Building base/height	KyFromAbove ground.f32 and KyFromAbove-derived height reference	Nearest-ground and city-plane fallbacks, height confidence, and the 7 m unmatched default are recorded rather than presented as surveyed values.
Routable road geometry	LFUCG Street Centerline	RDCLASS, posted speed, direction, route, and maintenance are retained. Pavement width is a documented class model because physical width is not published. OSM remains a lightweight context/semantic source, not the authoritative centerline.
Signal locations	Active records in LFUCG Traffic Signal	The current provenance download takes precedence over the bundled legacy LFUCG cache. Only when neither is available does the class/degree heuristic run, labeled class_heuristic. LFUCG does not publish phase timing here, so emitted fixed-time plans are labeled synthetic_fixed_time.
Pedestrian crossings	Explicit OSM highway=crossing, way highway=crossing, or way footway=crossing objects	OSM IDs and all tags are retained under ODbL. smooth_roads matches their representative points to LFUCG roads and models a candidate polygon from road bearing/class width. Explicitly unmarked/unknown records remain distinguishable from rendered markings. At an active LFUCG signal with no mapped crossing, a lower-confidence modeled_at_lfucg_signal crosswalk may be emitted; it is not claimed as as-built geometry.
Other controls	Published evidence only	The LFUCG centerline has no stop-sign field, so missing controls remain unknown rather than becoming invented stop signs.

tools.public_data paginates each ArcGIS FeatureServer, checks the advertised feature
count, writes atomically, hashes the result, and creates
web/data/sources/provenance.json with publisher, item/service URLs, query, source CRS,
last-edit time, retrieval time, record count, and SHA-256. fetch_osm_crossings separately
records the exact bbox, Overpass pages/mirrors, OSM base timestamps, full tags, WGS84 and
scene geometry, retrieval time, attribution, and ODbL terms. The generated roads,
signals, and building metadata carry the relevant source IDs and fallback labels forward.
The downloader also exposes bikeped, cameras, and streetlights for downstream
audits and placement tools; the current road/crossing bake does not pretend those
partial inventories are a comprehensive as-built crosswalk layer.

Reproduce the current layers

The order matters: osm_city establishes the AOI; KyFromAbove builds the elevation
grid; the legacy OSM building pass supplies a temporary measured-height reference; then
the LFUCG geometry replaces it. Existing installations that already have a
KyFromAbove-derived buildings.pack.json can skip the two bootstrap-building commands.

pip install "laspy[lazrs]" pyproj shapely scipy scikit-image numpy

# AOI and city elevation (requires the extracted/georeferenced campus manifest)
python -m tools.osm_city
python -m tools.ky_lidar --download-aoi
python -m tools.ky_lidar --build --heightmap

# Bootstrap a KyFromAbove-derived height reference only when no pack exists yet
python -m tools.build_city
python -m tools.pack_buildings

# Count-checked authoritative downloads + per-source provenance
python -m tools.public_data

# LFUCG road geometry, OSM explicit crossings, and source-aware signal model
python -m tools.lfucg_roads --out web/data/roads.json
python -m tools.fetch_osm_crossings
python -m tools.smooth_roads --city

# LFUCG footprints joined to measured heights; activate recoverably, then rebuild collision boxes
python -m tools.build_lfucg_buildings --activate
python -m tools.build_obb

The bootstrap build_city pass estimates a roof from the configured percentile
(85th by default) of in-footprint KyFromAbove class 1/6 returns above ground.f32,
falling back to explicit OSM height/levels when needed. The LFUCG builder does not
claim to re-measure each replacement polygon: it spatially transfers those heights,
uses a nearby reference within its bounded search when necessary, and labels an
unmatched default. That distinction is preserved in heightSourceCounts and every
building record.

build_lfucg_buildings first writes buildings.lfucg.pack.{bin,json}. With
--activate, it replaces the conventional one-draw-call buildings.pack.{bin,json}
and retains one recoverable *.pre-lfucg backup. Each building records the LFUCG source
ID, the height/ground source, and confidence. For a QA-first activation, run the builder
without --activate, inspect its summary (or run
python -m tools.verify_lfucg_buildings), then use --activate-existing. Always run
build_obb after changing the active pack so browser, server, and Gym collision geometry
remain in the same building order.

The LFUCG builder writes BPK2: a 20-byte count header, scene-space positions, the
complete wall+roof index stream, and a separate exact roof-only index stream. Roof
triangles must be fully covered by their source polygon (including concavities and
courtyards) and are wound upward. The viewer shares those exact vertices for nearby
walls and roofs; its city overview uses orientation-aligned two-triangle roof proxies
from buildings.obb.json, so the performance LOD cannot rotate a roof away from its
building. Legacy BPK1 packs remain readable.

Generated city counts intentionally are not frozen in this README: LFUCG, OSM, GTFS,
and KyFromAbove source editions change. Use sources/provenance.json, the stats or
scope blocks in generated JSON, and the builders’ count-checked summaries for the exact
artifact you trained against.

Performance contract: buildings remain one active packed draw call and road props use
merged or instanced geometry. Within 120 m, roads automatically switch to their exact
source-sampled ribbons; BPK2 buildings retain exact roofs and add nearby exact walls.
The frame-budget governor may shed labels, decorative props, LiDAR density, distant LOD,
and pixel density, but it does not change agent-facing roads, intersections, crossings,
or collisions. Resolution drops in 25%-of-native steps only inside the low-FPS guard band
(never below 50%), then probes back toward native after sustained headroom (roughly six
seconds from 50% under a stable 48+ fps load, while tolerating ordinary frame jitter).
Actual frame rate still depends on GPU, viewport,
photoreal tiles, camera streams, and enabled layers, so validate the target hardware.

The default adaptive packed view is the above-30-fps target path. Append
?roadDetail=1 to extend exact road rendering to 1.2 km or ?buildingDetail=1 for the
full building mesh during inspection; either override can reduce frame rate.

Export the complete static map to USD

After the generated city layers exist, export an Omniverse-compatible stage directly
from the exact active artifacts:

python -m tools.export_usd

This writes web/data/lexington.usd plus separate ground, building, road, intersection,
crosswalk, traffic-signal, static Lextran route/stop, and traffic-camera payload layers
beside it. Keep those payloads beside the root file and open lexington.usd in
Omniverse. The stage uses metres, Y-up coordinates, EPSG:32616 georeferencing metadata,
source provenance, feature IDs, and semantic class labels. The payload split keeps the
stage streamable and lets downstream training tools load only the layers they need. Use
python -m tools.export_usd --help to select layers, output path, precision, or omit
feature-ID primvars. USD output remains under gitignored web/data/ and is regenerated
from the public-data pipeline rather than committed. Runtime agents, live bus poses,
camera detections, and browser-local LoRaWAN plans remain dynamic; export a radio plan
from window.__twin.lorawan.toGeoJSON() when it needs to travel with a training run.

Licensing and redistribution

• LFUCG downloads are public but governed by the city’s custom
  Terms of use, not a blanket CC0/CC-BY grant;
  individual items can add terms. Review them before redistributing a derived pack.
• KyFromAbove/KYAPED data is public domain under its published
  distribution policy.
  Keep acquisition date, CRS, units, and vertical-datum metadata with derived elevations.
• OpenStreetMap is ODbL 1.0. Preserve
  “© OpenStreetMap contributors,” the copyright link,
  and applicable database share-alike obligations. The temporary height-reference build
  uses OSM footprints, so do not relabel the activated result as solely LFUCG-licensed
  without a license review.
• Optional KYTC layers retain the publisher’s disclaimer URL in provenance. Their
  coverage and collection date must be checked before treating them as surveyed truth.
• Keep the differently licensed source files and provenance records separate. Live
  Lextran and traffic-camera feeds are dynamic services with their own terms; they are
  not authoritative substitutes for the baked GIS layers.

Photorealistic basemap — Google 3D Tiles (opt-in)

The viewer can stream Google Photorealistic 3D Tiles (the same textured-mesh data
Google Earth uses, and the same source reference Cesium-ion twins pull from) directly
into the Three.js scene as an optional basemap — real building textures, trees, and
terrain under the live overlays (cameras, buses, agents, signals). It is rendered with
the vendored NASA-AMMOS 3d-tiles-renderer
(web/lib/3d-tiles-renderer.module.js, pinned to a build compatible with the viewer’s
three r160) and web/tiles3d.js.

It is fully self-contained: nothing is fetched and there is no per-frame cost until
the layer is enabled and a key is present, so the rest of the twin is untouched. Toggle
it under Photorealistic 3D (Google) in the panel.

You need a key. Google’s tiles require a Google Maps Platform API key with the
Map Tiles API enabled (it has a free monthly tier; restrict the key to your domain
in the Cloud console). Provide it any of these ways:

• .env (recommended): cp .env.example .env and set GOOGLE_MAPS_API_KEY=…. The
  file is gitignored; tools/twin_server.py loads it and serves the key to the viewer via
  /api/photoreal (it is never written into the repo or a tracked file); or
• paste it into the panel’s "Google Maps API key" box and click Use & save key — the
  server writes it to .env for you, so you only enter it once (it’s reused on restart and
  never re-prompted); or
• export it inline — GOOGLE_MAPS_API_KEY=… python -m tools.twin_server; or
• append ?gkey=YOUR_KEY to the viewer URL.

A Cesium ion token works too (it serves the same Google dataset as asset 2275207):
pass ?ionkey=…, or set PHOTOREAL_PROVIDER=ion with the token in GOOGLE_MAPS_API_KEY.

Viewing it. The photoreal basemap is on by default (loads with the rest of the map
on initial load; pass ?photoreal=0 to start with it off). The default is now real
elevation so roads/labels/traffic drape onto the photoreal terrain (pass ?flat=1 for
the old single-plane mode). Tiles are streamed live from the Map Tiles API at the highest
fidelity the data supports — the detail slider sets the target screen-space error in
pixels (lower = sharper, more tiles; default 4, drop to 1–2 for maximum detail or
raise it for performance) and LOD refines as you zoom in.

Draping overlays onto the mesh. Our overlays (road ribbons, lane markings, signals,
street labels, buses, and traffic-camera/YOLO cars) are baked at our DTM/LiDAR elevation
(NAVD88-ish), while Google’s mesh sits on the WGS84 ellipsoid — tens of metres higher here,
and the gap varies across the city (geoid + a terrain-source mismatch). So when the
photoreal layer is on, the viewer samples that vertical gap at the camera’s look-at point
each frame (a downward raycast against the mesh vs. our ground) and lifts the overlay
groups onto the Google surface, eased so panning across relief stays smooth. Without it the
overlays sink under the mesh. It’s automatic; it turns off (eases back to our own terrain)
when the photoreal layer is off or in ?flat=1 mode.

Tile caching. When served by tools.twin_server the viewer routes tile fetches through
a bounded on-disk cache proxy (/api/gtile → web/data/tilecache/, gitignored,
host-locked to tile.googleapis.com) so repeat sessions reuse already-streamed tiles
instead of re-fetching them. This is a transient performance cache, not an offline
mirror: it is size-capped (LRU-trimmed at TILECACHE_MAX_GB, default 4 GB) and
root.json is always fetched live to keep the session valid. Set TILECACHE_TTL_DAYS for
a finite expiry, or TILECACHE_MAX_GB=0 to disable the cap. Delete web/data/tilecache/
any time to clear it.

Alignment. The tiles arrive in geocentric ECEF; tools/fit_tiles_align.py bakes a
similarity transform (web/lib/tiles_align.json) from the exact pipeline projection
(pyproj UTM-16N, the same Projector the OSM/transit bakers use), so the photoreal mesh
lines up with the OSM extrusions to ~2 m RMS (≤5 m) horizontally across the whole
city. It absorbs the UTM grid convergence (~1.5°) and point scale automatically. The
vertical datum (NAVD88 ↔ WGS84 geoid, ≈ −33.5 m here) carries a few metres of residual;
nudge it live from the console:

window.__viewer.photoreal.calibrate({ dy: 3, dx: 0, dz: 0, yaw: 0, scaleMul: 1 });

Custom / Cesium-ion tilesets. window.__viewer.photoreal.loadTileset(url) loads any
same-origin or CORS-enabled tileset.json through the same alignment + render path — for
your own 3D-Tiles exports.

Reproduce / regenerate from a clone (the vendored bundle + align matrix are committed, so
this is only needed if you bump versions or move the georef anchor):

cd web && npm install && node lib/_vendor.mjs      # rebuild bundle + vendor DRACO decoder
python -m tools.fit_tiles_align                    # rebuild lib/tiles_align.json
python -m tools.verify_photoreal                   # headless (no key): regression + alignment + render
node tools/verify_tiles_align.mjs                  # offline alignment-matrix check
# with a key in .env (these hit the real Google API):
python -m tools.verify_photoreal_live              # real Google tiles fetch + render + vertical probe
python -m tools.verify_tilecache                   # cache proxy: download once, serve from disk

node lib/_vendor.mjs (npm run vendor:tiles) produces both web/lib/3d-tiles-renderer.module.js
and web/lib/draco/gltf/ (the DRACO decoder real Google tiles need), version-locked to the
pinned three. All three — bundle, decoder, and tiles_align.json — are committed, so a
plain clone runs without this step.

Performance note: the photoreal mesh is many draw calls (like any streamed 3D-Tiles
basemap), unlike the packed local city layers — which is why it is opt-in and off by
default. Enabling "hide our buildings & ground" (default) drops the now-redundant grey
extrusions so the overlays read cleanly on top of the mesh.

Viewer controls

Control	Action
Left mouse	Orbit
Right mouse	Pan
Scroll	Zoom
WASD	Fly
Q / E	Down / Up
Shift	4x speed

UI panel: layer toggles, terrain opacity, UV V-flip, point cloud budget slider,
wireframe mode, camera reset.

Flat world (default). The viewer pins terrain, roads, the city ground plane,
buildings, buses, agents, and camera markers to a single elevation (web/flat.js),
so moving vehicles always sit on the road instead of clipping through it at the
campus/city seam (the two elevation systems — campus LiDAR relief and the lower flat
city plane — used to disagree there). The campus relief is intentionally discarded in
this mode; append ?flat=0 to the URL to restore the real LiDAR terrain elevation
and per-surface draping.

Generated artifacts and exact counts

web/data/ is generated and gitignored, so a single README count would describe one
source snapshot rather than the artifact on your machine. Use the machine-readable
records that travel with each build:

• web/data/sources/provenance.json — advertised/downloaded feature counts, source
  edit and retrieval times, queries, URLs, CRS, byte size, and SHA-256 for public GIS.
• web/data/city.json — the exact lon/lat and scene bboxes for that GTFS-derived AOI,
  its city ground elevation, and lightweight OSM context.
• web/data/buildings.pack.json — active building count, source item/retrieval data,
  AOI scope, and per-building height/ground provenance and confidence.
• web/data/roads.json — LFUCG source metadata and build stats; each road retains its
  centerline source ID and routing attributes.
• web/data/signals.json — junctions plus crosswalkStats, control provenance,
  crosswalk provenance/confidence, and synthetic phase-plan labels.
• web/data/transit.json — the baked route/stop snapshot; live vehicles, arrivals, and
  alerts are reported separately by /api/transit/meta and its feed endpoints.

This keeps training manifests reproducible without silently treating a later LFUCG,
OSM, GTFS, or KyFromAbove refresh as the same dataset.

Digital twin — controllable signals + autonomous agents

The road network is built for autonomous-agent simulation: tools/smooth_roads.py
emits web/data/signals.json, a machine-readable model of derived junctions and
explicit mid-block crossings (approach legs, control provenance, stop-line coordinates,
available crosswalk polygons, signal phase groups, and labeled synthetic fixed-time
plans). The viewer ticks a deterministic signal state machine and
exposes it at window.__twin.signals, so an agent can ask "what is my light right
now and where do I stop" (getLegState / queryByPosition) and even drive the
lights (setOverride).

On top of that, web/agents.js adds controllable agents — spawn a car, truck,
robot, or drone at window.__twin.agents, drive it from your own code, and read
back a POV camera, live position (scene m / UE cm / UTM-16N), object
collision detection (you program the avoidance), and a ground/surface probe
that keeps ground vehicles on the road or terrain and tells you which one they’re
on. Full API + schemas for both in web/README.md; smoke-test the
agent sensors with python -m tools.verify_agents.

The twin also carries the real Lexington bus network. tools/twin_server.py proxies
the live Lextran GTFS-Realtime feed and the viewer animates the actual buses on the
map — with route lines, stops, predicted arrivals, and service alerts — all reachable
at window.__twin.transit (getVehicles / getNearestVehicle / getArrivals / …).
Buses are wired into the agent sensor bus too (sensors.transit), so an autonomous
agent can yield to or wait for a real campus bus. Because the network spans far past
the campus tiles, tools/osm_city.py lays down the rest of Lexington (OSM streets +
a ground plane) so every route has ground beneath it. Smoke-test the live layer with
python -m tools.verify_transit.

Radio survey and LoRaWAN placement

The Radio plan workspace supports manual, map-click, public-inventory-candidate, and
repeatable seeded-random gateway placement. Set any 100–2500 MHz frequency, antenna
height, spreading factor, and fade margin; the planner derives the link budget and builds
a three-dimensional range shell whose rays stop at known building volumes. Gateway
markers remain one instanced draw call and the merged coverage shell remains one draw
call, with the geometry work performed off the render thread.

The Radio survey scenario now owns a quadrotor at drone altitude rather than a car.
For scripted experiments, use window.__twin.lorawan.addRandom({ count, seed }),
setProfile({ frequencyMHz, heightM, spreadingFactor, fadeMarginDb }), telemetry(),
and toGeoJSON(). The same seed and source snapshot reproduce candidate sites; GeoJSON
exports include frequency, RF-model inputs, placement method, seed, and public-source
metadata. Coverage is a planning prediction, not a substitute for propagation tooling,
site permission, or a field survey.

Live traffic cameras

The twin also shows real traffic at the intersections. The City of Lexington
publishes a public traffic-camera inventory; tools/lex_cameras.py projects each input
camera’s lon/lat through the same UTM-16N → scene georef the rest of the twin uses and
keeps that published GPS point as the sensor origin, draped to the KyFromAbove ground
grid. It records a nearby signals.json intersection only as a semantic link; the point
is never displaced to manufacture a match. Read the generated file/provenance for the
exact camera snapshot rather than relying on a README count. The viewer
(web/cameras.js) draws teal markers with a nearby junction link and amber unmatched
markers, and lists them all in the Traffic cameras panel.

Click a marker (or a list row) to open that intersection’s real-time video in a
picture-in-picture panel — just the stream, no detection. The camera feeds are
tokenized HLS URLs the city re-signs every ~15 minutes, so they can’t be baked: like
the bus positions, tools/twin_server.py re-scrapes fresh URLs on demand and serves
them same-origin (/api/cameras/streams); the browser plays them directly with the
vendored hls.js (the camera origin sends permissive CORS, so no segment proxying is
needed). With no server running you still get the markers and the token-free snapshot
thumbnail; with it you get live video. Everything is reachable from the console at
window.__twin.cameras (list / getNearest / streamUrl / still).

Regenerate the mapping (e.g. if the city adds cameras) with python -m tools.lex_cameras
(reads TrafficStream’s cached cam_data.json; pass --scrape to pull a fresh list, or
--cam-data PATH to point at a specific file). The twin includes its own optional live
detector; ultralytics and OpenCV are imported only when detection is enabled.

Camera-detected cars (Phase 1 — geometry + spawn)

Work in progress toward spawning a twin car for each vehicle a camera sees. Each stream
is a 2×2 quad of four independent wide-angle views, so there’s no automatic camera→world
mapping; instead you calibrate it. Open a camera’s PiP, click Calibrate, then
click a point in the camera image and the matching spot in the 3D twin (4+ per quad) — the
viewer solves a per-(camera, quad) homography (web/homography.js, dependency-free,
Hartley-normalized) and reprojects the fit back onto the video to judge. Calibrations are
saved (version-controlled) under calibration/cameras.json via the twin server
(/api/cameras/calib). Then Spawn mode: click the video where a car is and a
kinematic car appears in the shared world at the mapped scene point, visible to every
viewer (netagents.js). Kinematic agents (/api/world/.../pose, kinematic:true on
spawn) carry no physics and auto-despawn after --kinematic-ttl seconds (default 5)
without a pose update, so a feed that stops cleans up after itself.

Phase 2 — live detector. tools/camera_detect.py closes the loop: a decoupled world
client that pulls one camera’s HLS, splits the 2×2 source, and runs all relevant views in
one batched YOLO26n call by default. It maps each vehicle’s tire point through the
per-view homography, rejects points outside the authored calibration hull/horizon, and
validates motor vehicles against the authoritative road surface before tracking or
spawning. Small calibration errors snap to the road edge; material off-road mappings are
rejected. Cross-view deduplication and nearest-neighbour association provide stable IDs,
motion-derived heading, and despawn-when-lost behavior.
It talks to the twin over HTTP only, so it runs wherever ultralytics + OpenCV live (e.g.
TrafficStream’s GPU venv) with no campus deps; the geometry + lifecycle core is
dependency-free and unit-tested. With the twin running and a camera calibrated:
python -m tools.camera_detect --camera LEX-CAM-052. Verified end to end on the live
Harrodsburg/Lakespur feed (real YOLO detections → scene points → tracked twin cars).
Both the external detector and the in-process PiP detector default to CUDA FP16 when the
installed PyTorch build reports a usable GPU; their telemetry reports the actual device,
inference time, decoded-frame age, and dropped-frame count.

Phase 3 — detection overlay + perf bounding. The detector relays the image-space
boxes it spawns cars from (/api/cameras/detections); the PiP’s Detect toggle polls
and draws them, class-coloured, on the quad — so you see this detection → this twin car.
Opening a camera’s PiP also signals it as the active camera (/api/cameras/active), so a
detector started with --follow-active only burns GPU on the camera someone’s actually
watching. The PiP reports detector rate, inference time, decoded-source age, mapped and
tracked counts, calibration-gated boxes, and road rejections so latency and mapping
quality are visible instead of hidden.

Spec + plan + tasks under specs/003-camera-detected-cars/ — feature complete through
Phase 3.

Multiplayer twin server — shared world over an API

window.__twin.agents is private to one browser tab. For a shared world — where
the twin runs on its own server and many scripts/users drive agents that everyone
sees — run the authoritative server:

python -m tools.twin_server        # viewer + world API + live buses on :8000

Run it as a module from the repo root (python -m tools.twin_server), not
python tools/twin_server.py — it imports the tools package, and running it as a
loose file breaks that import. One server now does it all on a single port: the viewer,
the world API (/api/world/*), the live bus proxy (/api/transit/*), and — with
--render — first-person agent cameras.

404/503 on /api/world/...? Something else is answering on :8000 — usually a
plain python -m http.server left running, which serves the viewer’s files but has no
world API. Stop it and run python -m tools.twin_server instead. To run a second server
on the same machine, give the twin another port: python -m tools.twin_server --port 8001
and point clients/browser at :8001.

It holds every agent in one place, ticks the physics (the same ackermann / differential
/ holonomic-drone kinematics as agents.js, with ground from the terrain heightmap and
collisions from the baked building OBBs), and exposes a small REST API. Agents spawned
by ANY client are visible to ALL of them — other scripts and any browser open on the
server (web/netagents.js renders the shared world in the Shared world (server)
panel section).

Drive it from Python with the dependency-free client (client/twin.py):

from twin import Twin
twin  = Twin("http://twin-host:8000", owner="alice")
drone = twin.spawn("drone", position=[0, None, 0])
drone.set_controls(move=[5, 1, 0])          # fly +X and climb
print(drone.state()["position"], drone.collisions())
for other in twin.agents():                 # every agent in the shared world
    print(other["owner"], other["type"], other["position"])
drone.stop(); drone.despawn()

The REST surface (all JSON, CORS-open): GET /api/world/state (everyone’s agents),
GET /api/world/agents/<id>, POST /api/world/spawn, POST /api/world/agents/<id>/{controls,driveTo,stop},
DELETE /api/world/agents/<id>, plus GET /api/world/{meta,nearest_building}.

Runtime traffic and scenario API

The viewer’s simulation console and external orchestrators use the same bounded
/api/sim/* surface. NPCs drive the active road graph (LFUCG-derived after the rebuild)
and consult the
server-side deterministic signal model, so browser observers and API agents see the
same traffic state.

Endpoint	Behavior
GET /api/sim/status	Active preset, pause/time scale, sim clock/frame, agent/traffic totals and seed, signal readiness, and tick CPU/overrun metrics.
GET /api/sim/scenarios	Preset catalog plus the current preset and enforced traffic/time-scale limits.
POST /api/sim/traffic	{ "count": 0..40, "seed"?: 0..4294967295 }; count-only calls resize the managed pool, while supplying a seed deterministically rebuilds it.
POST /api/sim/scenario	{ "name": "mobility_baseline" | "rush_hour" | "autonomy_eval" | "radio_survey" }; loads the preset from its fixed seed.
POST /api/sim/control	{ "paused"?: bool, "timeScale"?: 0.1..4.0 }.
POST /api/sim/reset	Rebuilds the active preset, or custom traffic setup, from its retained seed and resets the simulation clock.
POST /api/sim/clear	Stops and joins all in-process camera detectors, then removes every shared-world agent, including managed traffic and scenario-owned actors. The response includes detector stop counts.

The built-ins are mobility_baseline (12 NPC cars), rush_hour (32),
autonomy_eval (16 plus one scenario-owned ego car), and radio_survey (4 plus one
scenario-owned drone). Their seeds and descriptions are returned by the catalog rather
than duplicated in training code.

curl -s http://localhost:8000/api/sim/scenarios
curl -sX POST http://localhost:8000/api/sim/scenario 
  -H "content-type: application/json" -d '{"name":"autonomy_eval"}'
curl -sX POST http://localhost:8000/api/sim/traffic 
  -H "content-type: application/json" -d '{"count":20,"seed":77}'
curl -sX POST http://localhost:8000/api/sim/control 
  -H "content-type: application/json" -d '{"paused":false,"timeScale":2}'
curl -sX POST http://localhost:8000/api/sim/reset 
  -H "content-type: application/json" -d '{}'

Scenario loads and resets intentionally clear the shared world before rebuilding it;
do not call them while another user’s agent must survive. The server is CORS-open and
has no authentication. It binds to 0.0.0.0 by default, so use
--host 127.0.0.1 or an authenticated reverse proxy outside a trusted LAN. Verify the
control contract without loading city artifacts using
python -m tools.verify_sim_control.

/api/sim/clear can stop detector threads hosted by this server. A separate
python -m tools.camera_detect process or any other external producer is an independent
client and can spawn agents again after the clear; stop those producers separately when
the world must remain empty.

A runnable example — spawn a drone, fly a circuit, fly into a building until the
collision sensor fires, then stop — is in examples/drone_demo.py:

python -m tools.twin_server &                 # the twin
python examples/drone_demo.py --url http://localhost:8000   # a client script
# open http://localhost:8000/ in a browser to watch it in 3-D

First-person cameras + vision navigation (YOLO)

The server can also produce a first-person video feed for every agent, so a script
can drive by what the agent sees instead of by ground-truth state. The server has no
renderer of its own, so --render attaches a headless browser (the viewer) as an
internal render service and serves JPEG frames:

python -m tools.twin_server --render          # adds /api/world/agents/<id>/camera

From a script, agent.camera_image(w, h) returns the frame as a PIL image. The
car / truck / robot examples feed it to the smallest YOLO model (yolov8n)
and steer to avoid what it detects — e.g. it flags the campus traffic-signal heads as
COCO traffic light and vehicles in the aerial ground imagery as car:

pip install -r examples/requirements.txt      # ultralytics (smallest YOLO) + torch (CPU ok)
python -m tools.twin_server --render &
python examples/car_demo.py                    # or truck_demo.py / robot_demo.py
# open http://localhost:8000/ to watch them drive themselves

All three spawn into the same shared world, so you can run them together (and watch in a
browser) and they’ll see and bump into each other. client/twin.py exposes the feed as
agent.camera() (JPEG bytes) / agent.camera_image() (PIL); examples/yolo_drive.py
holds the shared camera→YOLO→controls loop.

Gym environment (campus_gym)

For training/evaluation, the simulation core (tools/twin_server.World/Agent) is
also wrapped as a synchronous, headless gym environment — it advances only inside
step() (no server, no browser, no real-time clock), so it runs faster than real time
and is reproducible. The real-time REST server stays for interactive/multi-client use;
this is its training-shaped twin.

import gymnasium as gym, campus_gym          # registers Campus-v0, CampusDrone-v0, ...
env = gym.make("Campus-v0")                   # single agent (Gymnasium)
obs, info = env.reset(seed=0)
obs, reward, terminated, truncated, info = env.step(env.action_space.sample())

Task (Tier 0): drive a car/truck/robot/drone to a seeded goal on campus
without crashing into a building or leaving the map.

• observation Box(13): ego kinematics + nearest-building (ego frame) + goal (ego frame)
• action: ground [accel/brake, steer] ∈ [-1,1]; drone [vx,vy,vz] ∈ [-1,1]
• reward: progress toward the goal − collision/off-map penalties − small time cost
• terminated reached goal / crashed / off map; truncated at max_episode_steps

Multi-agent is exposed as a PettingZoo Parallel env (every agent acts each step;
dict-keyed obs/reward/terminated/truncated/info; agents collide with each other):

from campus_gym import CampusParallelEnv
env = CampusParallelEnv(agent_types=("car", "truck", "drone", "robot"))
obs, infos = env.reset(seed=0)
obs, rewards, terms, truncs, infos = env.step({a: env.action_space(a).sample() for a in env.agents})

Read-only world data (terrain heightmap + building AABBs) is loaded once and shared
across env instances, so it vectorises (gymnasium.vector.SyncVectorEnv([...])).

Language-conditioned navigation + evaluation (the agentic layer)

CampusNav-v0 (and CampusNav{Car,Truck,Robot,Drone}-v0) turns the twin’s real
named entities — Lextran bus stops and named campus streets — into goals specified
in natural language. Each episode hands the agent an instruction grounded in a real
coordinate, so an LLM/VLM agent reads info["instruction"]:

env = gym.make("CampusNav-v0")
obs, info = env.reset(seed=0)
info["instruction"]   # e.g. "drive to the Transit Center" / "navigate to Pennsylvania Avenue"

Supporting pieces:

• Configurable reward — a sum of named, weighted terms (CampusEnv(reward_weights=...)),
  with the per-term breakdown in info["reward_terms"] for transparent shaping/ablation.
• Metrics (campus_gym.eval) — evaluate(env, policy, episodes) reports success rate,
  SPL (success weighted by path length, using the road-graph shortest route), collision
  rate, and mean return.
• Record + deterministic replay (campus_gym.record) — log an episode to JSONL
  (seed + actions + per-step reward + the instruction) and replay it exactly.

python examples/gym_nav_eval.py --type car --episodes 10   # instructions + SPL + replay

NPC traffic + signals, scenarios, vectorization, training

• Sensable traffic — CampusTraffic-v0 / CampusEnv(npc_traffic=N, signals=True) adds
  NPC cars that drive the real road graph (IDM car-following + red-light stopping) and the
  deterministic traffic signals (ported from the viewer into the authoritative World), so
  the agent perceives and must yield to them — the observation grows to 18 dims with nearest-
  vehicle + signal-ahead features. (Previously signals/transit lived only in the browser.)
• Scenarios + domain randomization (campus_gym.scenarios) — a declarative Scenario
  spec + registry (make_scenario("campus_traffic")), train_test_seeds() for held-out
  evaluation, and domain_random=True to jitter agent dynamics per reset (sim-to-real).
• Vectorization — SyncVectorEnv and AsyncVectorEnv (use the picklable campus_gym.make_env).
• RL training — examples/train_ppo.py trains Stable-Baselines3 PPO and evaluates it
  on held-out seeds before vs. after (return improves as it learns to drive to goals).

python examples/train_ppo.py --timesteps 50000 --n-envs 6      # pip install stable-baselines3

Verify the whole stack (Gymnasium check_env, PettingZoo parallel_api_test, seeded
determinism, Sync+Async vectorization, named/language goals, NPC traffic + signals, eval/SPL,
scenarios + domain randomization, record+replay) with python -m tools.verify_gym. This
covers Tiers 0-3 of the agentic-gym roadmap end to end. Remaining stretch items: importing
real traffic datasets, richer scenario authoring, and visual (camera) observations for the
trainer (the camera feed exists via the server’s --render, but the gym defaults to fast
state-vector observations).

---

Imported from gh:Kentucky-Open-Science/UKy-Campus-LIDAR. Source last updated 2026-07-18. Synced 2026-07-27.