Reconstruction Scale/Pose Mismatch: Predicted Poses and Intrinsics Not Aligning with Ground Truth

[LouisBdg]

Hello, thank you for this great repository.
I’m tracking cameras and reconstructing a scene using this repository. I have ground truth poses and intrinsics for each image. When I run inference with only images, the point cloud looks reasonable but is incorrectly scaled. Adding intrinsics doesn’t change the result, but adding poses improves the scale—however, the point cloud becomes incoherent and unrecognizable. To find where this problem comes from, I looked at input/output difference in poses and intrinsics.
I plotted the poses in 3D:
With images only: Point cloud is scaled incorrectly (4x too large). Predicted poses are in green, measured ones in blue.

With images plus intrinsics: Result is almost identical to image-only.

With images, intrinsics and poses: Scale improves, but is still incorrect, the point cloud is noisy and doesn’t match the scene, it seems like the pointclouds related to each input images duplicate the scanned object in different but nearby positions. My issue seems close to the #33 one.

About intrinsics:
The predicted intrinsics (fx, fy, cx, cy) are significantly different from the provided values. And this difference is not completely correlated to the reshape process of images (2048x1536) -> (518, 392) as seen in the issue #38 .
One of the differences is that input cx and cy are not centered while the predicted intrinsics are centered. I’ve seen a similar problem in issue #67 but with no solution for me.

Do somebody know where my errors come from? Is it normal to have predicted poses and intrinsics that differs that much from inputs? As somebody managed to have a satisfying result with poses and intrinsics as inputs?
Thank you for your help!

[shaashou]

Have you resolved these issues?

[Nik-V9]

Hi, thanks for testing MapAnything!

The quality of the pass through when using pose inputs depends on the testing domain since the model expects the pose scale to align with its internal metric scale prior (based on training data). For example, MapAnything has an issue with toy scale objects and aerial domain since they are not featured in the metric-scale training set.

When observing suboptimal performance using poses as input (the same case for COLMAP or up to scale poses as input), I would recommend testing out ignore_pose_inputs=True in the inference API. For example, you can see the effect of this flag here: #9 (comment)

[shaashou]

Hi, thanks for testing MapAnything!

The quality of the pass through when using pose inputs depends on the testing domain since the model expects the pose scale to align with its internal metric scale prior (based on training data). For example, MapAnything has an issue with toy scale objects and aerial domain since they are not featured in the metric-scale training set.

When observing suboptimal performance using poses as input (the same case for COLMAP or up to scale poses as input), I would recommend testing out ignore_pose_inputs=True in the inference API. For example, you can see the effect of this flag here: [#9 (comment)](#9 (comment)

Hi, thanks for testing MapAnything!

The quality of the pass through when using pose inputs depends on the testing domain since the model expects the pose scale to align with its internal metric scale prior (based on training data). For example, MapAnything has an issue with toy scale objects and aerial domain since they are not featured in the metric-scale training set.

When observing suboptimal performance using poses as input (the same case for COLMAP or up to scale poses as input), I would recommend testing out ignore_pose_inputs=True in the inference API. For example, you can see the effect of this flag here: #9 (comment)

Hello, is it possible to obtain the same depth information as the actual value if I only input a single viewpoint? I've observed that the camera position parameter isn't important if it's a single viewpoint; I only plan to input RGB, camera intrinsics, and depth map.

[Nik-V9]

Hi @shaashou yes that should be possible.

I would recommend trying both True and False for the ignore_depth_scale_inputs flag.

[shaashou]

I will try your method, and if I encounter any problems again, I hope to receive your response

[roboyike]

My input image is 720x1280. How should I process it? No matter what I do now, I can't infer my correct intrinsic parameters.

Hi @shaashou yes that should be possible.你好，是的，这应该是可行的。

I would recommend trying both True and False for the ignore_depth_scale_inputs flag.我建议 ignore_depth_scale_inputs 旗时可以同时尝试“真”和“假”两个选项。

[Nik-V9]

Hi @roboyike are you talking about the intrinsics pass through? Note that the model requires 518 longest side images as input.

[shaashou]

Hi @roboyike are you talking about the intrinsics pass through? Note that the model requires 518 longest side images as input.

I am currently also using 720x1280 resolution RGB images with the actual camera's intrinsic parameters. Should I convert my images to 512x294x3 format before feeding them into the preprocess_input function?

[Nik-V9]

The model inference takes care of the resizing and the output will correspond to the resized input. You can check the shape of the predictions or viz RGB for resized shape. Please see: #38

[shaashou]

The model inference takes care of the resizing and the output will correspond to the resized input. You can check the shape of the predictions or viz RGB for resized shape. Please see: #38模型推理过程会自动处理图像大小调整，输出结果将与调整后的输入图像相对应。您可以查看预测结果的形状，或者查看调整后图像的 RGB 值。请参阅： #38

I noticed this issue and found that it seems to eventually predict correctly, but my direct input of RGB and camera intrinsics, fx, fy, has significant differences, while cx and cy cannot be predicted correctly at all

[shaashou]

@Nik-V9

[shaashou]

The model inference takes care of the resizing and the output will correspond to the resized input. You can check the shape of the predictions or viz RGB for resized shape. Please see: #38

Hello, the first image shows the parameters of my data after being processed by the preprocess_inputs function, but after inference, the output does not match at all, just like the second image.

[shaashou]

These are all my code and the images I tested

demo_images_multi_inference.py

Before the test, I had completed the eye-on-hand calibration and obtained the camera intrinsic matrix. I very much hope to receive your reply

[shaashou]

The model inference takes care of the resizing and the output will correspond to the resized input. You can check the shape of the predictions or viz RGB for resized shape. Please see: #38模型推理过程会自动处理图像大小调整，输出结果将与调整后的输入图像相对应。您可以查看预测结果的形状，或者查看调整后图像的 RGB 值。请参阅： #38

I solved this problem; it was caused by an incorrect setting of my camera's internal parameters. Thank you for your patient explanation. @Nik-V9