# IMG-3D / Browser Reconstruction Lab

Live: https://hagop.io/img-3d/
Source: https://github.com/krelsys/hagop.io/tree/main/img-3d

This is a static, mobile-first and client-side experimental 3D scanning app in the Flagship Applications section. It supports live camera keyframes, local video keyframe sampling, and ordered still-photo batches. Captured media is not sent to a scan server.

## What the reconstruction really does

IMG-3D estimates relative depth using Depth Anything V2 Small and approximates 3D alignment across multiple keyframes with image matching and RANSAC. The updated live tab can optionally use device orientation as a cross-check on matched view rotation. Gyroscope orientation alone does not measure camera translation or object scale. Each accepted view contributes a separate colored triangle-grid surface. It does NOT perform calibrated photogrammetry, conventional structure-from-motion with bundle adjustment, dense metric multi-view stereo, real-time SLAM, volumetric TSDF fusion, loop closure, or watertight mesh reconstruction.

Inferred depth is scale-free and independently normalized for each view. Alignments can be imprecise, and even registered meshes can contain overlapping surfaces, holes and distortions. The live on-camera grid is a last-captured-keyframe reference, NOT continuously tracked AR. It cannot produce trustworthy measurements or production-ready 3D prints by itself.

## Pipeline

1. Input from getUserMedia camera, local image file imports, or local HTML video decoder.
2. Luminance and Laplacian-based sharpness diagnostics for live capture and pre-reconstruction rejection.
3. Browser inference with onnx-community/depth-anything-v2-small (Apache-2.0), Transformers.js 3.x and ONNX Runtime. WebGPU fp16 is attempted, with WASM q8 fallback.
4. Custom high-contrast keypoints and 160-bit BRIEF-like local descriptors with Hamming-distance ratio matching.
5. Between frames, median depth ratios at matched pixels estimate bounded relative depth scale continuity. This reduces (but cannot eliminate) per-frame depth normalization drift.
6. Pairwise 3D RANSAC rigid registration using depth-lifted matched features and Horn quaternion estimation. When a camera orientation reading is available at BOTH captures, a second pose estimate uses the relative camera rotation plus robust median translation from visual/depth correspondences. It is selected only when it agrees with visual inliers and residuals; optical registration remains the fallback. Low-inlier views and large movements are rejected.
7. Indexed RGB surface grids with discontinuity filtering, layered in a Three.js scene. Opacity, mesh lines and orbit controls.
8. Exports: binary GLB with vertex colors, binary PLY with vertex colors, OBJ with non-standard vertex RGB extension (viewer support varies), and binary STL without colors. Coordinate units are arbitrary.

## Recommended capture

Scan a stationary, matte subject under even light. Keep rich, non-repeating visible surface texture and 65–80% overlap between successive views. Move the phone slowly in small angular increments instead of jumping to a distant opposite side. For live scanning, enable Phone Gyro and allow motion permission. Aim for roughly 5–25° device rotation and low angular speed between keyframes; keep visual overlap high and watch the live feature matching HUD. A gyroscope cannot measure phone position, so rotate AND physically orbit around the object rather than spinning the phone at one spot. For photos, arrange the files in the desired orbital capture order before adding them. Video extraction samples frames by a user-specified interval. If alignment fails, recapture an intermediate view; do not force incompatible frames.

## Performance and compatibility

- No server or paid API account. JavaScript libraries and depth weights download from third-party hosts at first use. The source media is processed on-device.
- First depth model download: tens of MB. Working memory can be significantly higher.
- Maximum 32 accepted scan keyframes, and 32 queued imported images.
- GPU inference is serialized to avoid out-of-memory issues; camera preview, quality checks and 3D rendering can run concurrently.
- Works best on modern Chrome/Edge with secure origin and functioning WebGL2. WebGPU may be unavailable. WASM fallback can be very slow on low-power phones.
- Live camera permission is required. Mobile motion permission can be requested separately with ENABLE PHONE GYRO; without it the system continues using vision only. Android browsers may deliver sensor events automatically after enabling, while iOS requires permission from a user gesture. Orientation samples are attached only to live keyframes, not retroactively to uploaded videos or photos. Video formats must be decodable by the browser.
- JavaScript sources: index.html, app.js, engine.js, sensors.js. A reproducible synthetic-motion/geometry test and Chromium mobile UI smoke test run under GitHub Actions.
- This is an advanced experimental baseline, not a measurement instrument or a full professional photogrammetry replacement.

## Libraries and attributions

- Depth Anything V2 Small model: https://huggingface.co/depth-anything/Depth-Anything-V2-Small (Apache-2.0)
- Browser ONNX model conversion: https://huggingface.co/onnx-community/depth-anything-v2-small
- Transformers.js: https://github.com/huggingface/transformers.js (Apache-2.0)
- Three.js: https://github.com/mrdoob/three.js (MIT)

## Meaningful next upgrades

For reliable true multi-view photogrammetry: calibrated camera intrinsics; robust SIFT/SuperPoint matching; globally optimized SfM camera poses; loop closure; camera/depth scale consistency; confidence-weighted dense multi-view reconstruction or TSDF integration; mesh repair and automatic UV-texture atlas baking. None of those have been claimed as implemented in this release.

## Gyro-assisted capture limitations

- DeviceOrientationEvent reports orientation only; DeviceMotionEvent rotationRate improves motion-speed feedback when available. Neither provides reliable phone translation.
- Orientation estimates can be stale, subject to permissions, screen orientation and drifting calibration. A gyro pose prior is **never blindly imposed**: it must pass a visual residual/inlier check.
- The percentage shown as "VISUAL MATCH" is matched-keypoint ratio, **not** true image-area overlap; it is a cue, not a calibrated measurement.
- Auto keyframe capture favors stable, nonduplicate, feature-overlapping views. The "Allow low-quality keyframes" override does not make a wrong pose correct.
- Multi-view reconstruction remains approximate and no version of this code implements metrology-grade SfM or camera translation from inertial sensors.