PanoSharp

PanoSharp aligns and merges two 3DGS plys from single-image reconstructions using Apple's SHARP model. It provides a single unified CLI that handles both real photographs and AI-generated images. It takes the assumption from Sharp that the images are taken from the same camera with the same intrinsics.

Photos:

Source X	Source Y	Merged

Gemini-generated images:

Gemini X	Gemini Y	Merged

Installation

# Clone the repository
git clone --recurse-submodules https://github.com/Constannnnnt/PanoSharp.git
cd PanoSharp

# Install dependencies
uv sync

# Install SHARP (submodule)
uv pip install -e third_party/ml-sharp

Requirements: Python 3.13+, PyTorch 2.8+, CUDA (recommended). Tested on Windows 11 with RTX 3070.

Quick Start

Align Real Photographs

python main.py \
    --src photo1.jpg \
    --tgt photo2.jpg \
    --type real \
    --output outputs/

Align AI-Generated Images

python main.py \
    --src ai_image1.png \
    --tgt ai_image2.png \
    --type ai \
    --output outputs/

Use Pre-existing PLY Files (Skip SHARP)

python main.py \
    --src_ply outputs/src.ply \
    --tgt_ply outputs/tgt.ply \
    --src_img src.jpg \
    --tgt_img tgt.jpg \
    --type real \
    --output outputs/

Parameters

Image Type (--type)

Type	Strategy	Best For
real	Sim3 + RBF	Camera photographs
ai	Homography + RBF	AI-generated images

Non-Rigid Warping

Parameter	Default	Description
--smoothing	0.3	RBF smoothing (higher = smoother)
--warp-strength	1.0	Warp intensity (0 = rigid only)
--falloff	0.0	Distance falloff (0 = no falloff)

Recommended Configurations

Conservative (less distortion):

--smoothing 1.0 --warp-strength 0.5 --falloff 0.5

Aggressive (tighter fit):

--smoothing 0.1 --warp-strength 1.0 --falloff 0.0

Algorithm Overview

Real Image Alignment (Sim3 + RBF)

1. Feature Matching: SIFT with ratio test and RANSAC
2. Depth Rendering: gsplat-based Gaussian depth rendering
3. Sim3 Estimation: Umeyama algorithm with RANSAC
4. Depth Affine Correction: f(z) = a + b/z
5. GMM/ICP Refinement: Point-to-plane with color gating
6. Non-Rigid Warping: Thin-plate spline RBF interpolation

AI Image Alignment (Homography + RBF)

1. Feature Matching: SIFT with geometric verification
2. Homography Estimation: RANSAC-based 2D homography
3. Scale Estimation: Median depth ratio at matched points
4. Homography Warping: 3D → 2D → warp → 3D
5. 2D RBF Smoothing: Smooth residual correction in image space

API Usage

from pathlib import Path
from panosharp import PanoSharpPipeline, PipelineInput, ImageType

# Create pipeline with default components
pipeline = PanoSharpPipeline.create_default(device="cuda")

# Configure input
input_config = PipelineInput(
    src_image_path=Path("photo1.jpg"),
    tgt_image_path=Path("photo2.jpg"),
    output_dir=Path("outputs/"),
    image_type=ImageType.REAL,
    smoothing=0.5,
    warp_strength=1.0,
)

# Run pipeline
result = pipeline.run(input_config)

if result.success:
    print(f"Aligned: {result.aligned_ply_path}")
    print(f"Merged: {result.merged_ply_path}")

License

See LICENSE file. Note that this project is a wrapper around Apple's SHARP model, which is subject to their license. gsplat is also subject to its own license.

Disclaimer

This project is provided as-is and is not affiliated with Apple or any other company. Use at your own risk.