feat(pipeline): Hybrid SDF→Gaussian pipeline for fast rendering of SDF geometry

[hummat]

Summary

Add a two-stage pipeline that trains an SDF model for high-quality geometry, then initializes 3D Gaussians on the extracted surface for real-time rendering and optional material decomposition.

Context: Recent methods like NeuSG, GSDF, and HybridNeRF demonstrate that combining SDF geometry with Gaussian rendering achieves both high-quality surfaces and real-time performance. Mini-mesh already has both halves (SDFStudio + Nerfstudio/splatfacto) but no bridge between them.

---

Motivation

Current mini-mesh pipeline forces a choice:

• SDF path → Best geometry (Neuralangelo, NeuS-facto), but slow training (hours) and no real-time rendering
• Gaussian path → Real-time rendering, fast training, but poor geometry (no mesh, just splats)

A hybrid pipeline gets both: SDF-quality meshes + GS-speed rendering. This is exactly what methods like NeuSG, GSDF, and HybridNeRF do internally, but we can achieve it at the pipeline level by chaining existing tools.

Use cases

1. Quality mesh + real-time viewer: Train NeuS-facto → extract mesh → initialize Gaussians on mesh surface → fine-tune splatfacto for real-time viewing
2. Material decomposition: SDF geometry provides clean normals → initialize GaussianShader/GS-IR on mesh → decompose materials with known geometry
3. Faster convergence: SDF-initialized Gaussians converge faster than random/SfM init because they start on the surface

---

Proposed Design

Pipeline extension

[Existing]                              [New]
SDF train → mesh extraction ──────────→ Gaussian init on mesh → GS fine-tune → splat/mesh export
                              ↓
                         (mesh.ply)

New stage between EXPORT and a second TRAIN:

Step 1: Surface sampling

Sample points on the extracted SDF mesh to create a dense point cloud for Gaussian initialization:

# Sample N points on mesh surface with normals
python -c "
import trimesh
mesh = trimesh.load('mesh.ply')
points, face_idx = trimesh.sample.sample_surface(mesh, count=100000)
normals = mesh.face_normals[face_idx]
# Write as COLMAP-compatible point cloud or transforms.json init
"

Step 2: Gaussian initialization

Initialize splatfacto from the sampled point cloud instead of sparse SfM:

• Gaussian centers = sampled surface points
• Gaussian orientations = aligned to mesh normals (shortest axis = normal)
• Gaussian scales = based on local point density
• This replaces the default SfM sparse init

Step 3: Fine-tune with geometric constraints

Train splatfacto (or 2DGS/DN-Splatter) with:

• Surface-initialized Gaussians
• Optional normal consistency loss (normals already known from SDF)
• Fewer iterations needed (geometry already correct, just optimizing appearance)

CLI interface

# Option A: Explicit two-stage
./run.sh --model neus-facto ...           # Stage 1: SDF
./run.sh --model splatfacto --init-from-mesh   # Stage 2: GS

# Option B: Single hybrid command
./run.sh --model hybrid-neus-splat ...    # Chains both automatically

Integration points

train.sh:

• New --init-from-mesh flag that samples the mesh and creates a dense point cloud
• Pass to nerfstudio as initial point cloud (nerfstudio already supports custom init via --pipeline.datamanager.dataparser.load-3D-points)

export.sh:

• After GS fine-tuning: export both splat (for real-time) and mesh (via TSDF/Poisson from the refined Gaussians)

run.sh:

• If --model hybrid-*: chain SDF train → mesh extract → GS init → GS train → export

---

Technical Details (from literature)

NeuSG approach (arXiv:2312.00846)

• Uses 3DGS to provide multi-view consistent pseudo-depth/normal supervision for NeuS
• Gaussians constrained near SDF zero-level set
• Joint optimization, but conceptually: GS guides SDF geometry

GSDF approach (arXiv:2403.16964)

• Dual-branch: SDF branch for geometry, GS branch for rendering
• Mutual supervision: SDF provides depth/normal priors to GS, GS provides photometric signal to SDF
• Key detail: GS densification guided by SDF gradient magnitude

HybridNeRF approach (arXiv:2312.03160)

• Spatially-varying β(x) field: >95% of scene rendered as surfaces (sphere tracing), rest as volume
• Distance-adjusted Eikonal loss: 1/d² weighting for unbounded scenes
• Two-stage: global β → per-voxel β(x) with distillation to small MLP

Practical simplification for mini-mesh

Rather than joint optimization (complex, requires deep framework changes), use sequential chaining:

1. Train SDF (existing pipeline) — get best geometry
2. Extract mesh (existing pipeline) — get clean surface
3. Sample mesh → initialize Gaussians → fine-tune GS (new) — get fast rendering

This is simpler than NeuSG/GSDF and leverages existing tools without modifying either framework.

---

Alternatives Considered

1. Joint SDF+GS training (NeuSG/GSDF style): Requires deep integration between SDFStudio and Nerfstudio training loops. Rejected for now: too invasive, sequential chaining gets 80% of the benefit.

2. GS-only surface methods (feat(pipeline): Support Gaussian-based surface reconstruction methods #13): Complementary, not a replacement. GS surface methods (2DGS, SuGaR) produce good meshes fast, but SDF methods still produce the highest-quality geometry for challenging cases (thin structures, textureless regions).

3. Export SDF to alternative renderers (e.g., nvdiffrec for baked rendering): Partially covered by feat(docker): Add nvdiffrast to Docker image for faster texturing #7/feat(export): PBR-ish texture extraction and BRDF-aware training #11. Doesn't give real-time GS rendering.

---

Tasks

• Implement mesh → dense point cloud sampling script (scripts/sample_mesh.py)
• Add --init-from-mesh flag to train.sh for splatfacto init
• Wire point cloud init through nerfstudio's dataparser
• Add --model hybrid-neus-splat (or similar) to run.sh for chained execution
• Test convergence: SDF-init splatfacto vs SfM-init splatfacto (quality + speed)
• Add config files (config/hybrid-neus-splat*.sh)
• Document workflow in docs/methods_and_models.md
• Optional: Add normal consistency loss during GS fine-tune (using SDF-derived normals)

---

References

Method papers

• NeuSG — GS-guided neural implicit surface reconstruction
• GSDF — Dual-branch SDF+GS mutual supervision
• HybridNeRF — Adaptive surface/volume rendering

---

Related Issues

• feat(pipeline): Support Gaussian-based surface reconstruction methods #13 — GS-based surface reconstruction (complementary: GS-only path)
• feat(export): PBR-ish texture extraction and BRDF-aware training #11 — PBR texture extraction (hybrid mesh + GS enables material decomposition)
• feat(docker): Add nvdiffrast to Docker image for faster texturing #7 — nvdiffrast Docker (enables GPU texture baking on SDF mesh)
• feat(pipeline): End-to-end validation with ground truth meshes #5 — E2E validation (compare hybrid vs SDF-only vs GS-only quality)