Bad output point cloud when passing depth as input.

[chrisoffner3d]

I have what I assume to be a related issue to #33 where the resulting point cloud gets very weird if I give MapAnything all geometric inputs, including depth maps.

Specifically, I perform inference as shown below, gradually "switching on" different geometric inputs. Results look fine and fairly similar for the variants no_geometric_inputs, intrinsics, and intrinsics + pose, but for intrinsics + pose + depth the point cloud looks really weird (see below). Any idea what might be the problem here?

Ground truth

apartment_gt_new.webm

MapAnything without any geometric inputs

preds = model.infer(
  processed,
  memory_efficient_inference=False,
  use_amp=True,
  # Ignore inputs that should not be used by the model.
  ignore_calibration_inputs=True,
  ignore_depth_inputs=True,
  ignore_pose_inputs=True,
  ignore_depth_scale_inputs=True,
  ignore_pose_scale_inputs=True,
)

mapanything_no_geometric_inputs.webm

MapAnything with intrinsics, pose, and depth inputs

preds = model.infer(
  processed,
  memory_efficient_inference=False,
  use_amp=True,
  # Ignore inputs that should not be used by the model.
  ignore_calibration_inputs=False,
  ignore_depth_inputs=False,
  ignore_pose_inputs=False,
  ignore_depth_scale_inputs=False,
  ignore_pose_scale_inputs=False,
)

mapanything_all_geo.webm

[Xiaozi0]

Same problem in the eth3d/courtyard, when add depth the result is worse. Why did this happen?

[Nik-V9]

Hey @chrisoffner3d, thanks for trying it out! Based on the visual you shared, this seems to be some ray depth vs z-depth issue.

Can you provide more info on whether the processed data is being passed in according to the guidelines mentioned here: https://github.com/facebookresearch/map-anything?tab=readme-ov-file#multi-modal-inference

[Nik-V9]

Hi @Xiaozi0 thanks for testing it! I don't seem to see an issue on my end with ETH3D courtyard both in terms of quantitative benchmarking and qualitative visualizations. I find that the predictions become more precise and aligned when I pass in the depth as input.

I wonder if this could be an issue with the input preprocessing? Are you ensuring that your version of ETH3D is undistorted (a common lapse)?

Using images only as input:

Using images, intrinsics, pose and sparse depth as input:

GT for reference:

Quantitative benchmarking numbers on ETH3D courtyard using 32 views (sampled 10 different combinations - left is images only and right is the one where images, depth & cameras are provided as input):

[chrisoffner3d]

Based on the visual you shared, this seems to be some ray depth vs z-depth issue.

My depth maps (from Blender) are originally depth-along-ray but I do convert them to z-depth before MapAnything inference. When I visualize the difference between the original and z-depth, the differences increase with ray angle distance from the optical axis, as I'd expect.

# Build raw views (one dict per view).
views_raw = []
for i in range(num_frames):
  # Convert "depth along ray" to "Z-depth" format expected by MapAnything.
  depth_z = geometry_utils.depth_along_ray_to_z_depth(
    gt_bundle.camera_intrinsics[i],
    gt_bundle.depths[i],
  )
  views_raw.append(
    dict(
      img=gt_bundle.colors[i],
      intrinsics=gt_bundle.camera_intrinsics[i].astype(np.float32),
      depth_z=depth_z.astype(np.float32),
      camera_poses=gt_bundle.camera_poses[i].astype(np.float32),
    )
  )

# Initialize model.
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = MapAnything.from_pretrained("facebook/map-anything").to(device)

# Preprocess.
processed = image.preprocess_inputs(
  views_raw,
  resize_mode="fixed_mapping",
  resolution_set=518,
  norm_type=model.encoder.data_norm_type,
  patch_size=model.encoder.patch_size,
)

# Perform inference.
preds = model.infer(
  processed,
  memory_efficient_inference=False,
  use_amp=True,
  # Ignore inputs that should not be used by the model.
  ignore_calibration_inputs=False,
  ignore_depth_inputs=False,
  ignore_pose_inputs=False,
  ignore_depth_scale_inputs=False,
  ignore_pose_scale_inputs=False,
)

Can you provide more info on whether the processed data is being passed in according to the guidelines mentioned here: https://github.com/facebookresearch/map-anything?tab=readme-ov-file#multi-modal-inference

Is there any other aspect of the guideline you think I might be missing, other than Z-depth vs. depth along ray?

[Nik-V9]

Ah I see, I don't see anything wrong in the preprocessing. I know this is wrong but have you tried to use the depth inputs without converting them to z depth directly?

The other suggestion would be to try ignore_depth_scale_inputs and ignore_pose_scale_inputs to True. The geometric inputs can be used suboptimally if their scale doesn't align with the model's internal prior for metric scale.

Do the images have any sort of distortion?