Use depth image as additional input

[xuxw98]

Thanks for your great project!

The paper mentioned that SAM 3D can optionally take in point map as input. I notice in Inference class it supports pointmap, but there is no guidance on how to format the pointmap input.

In my case, I have a RGB-D image where the depth and RGB image are well aligned with known camera intrinsics. Could you please help me with more guidance on how to process the depth image to acquire the pointmap? Thank you very much!

[gleize]

Hi @xuxw98,

Regarding the exact pointmap shape and format, you can set a breakpoint here, run the demo.py file and figure out what shape / format it is from there. A pointmap encodes the x, y, z coordinates for each pixel, but I do not remember the exact dimension order.

Regarding the "depth-to-pointmap" conversion, as you mentioned, you would need the camera depth intrinsics to unproject each pixel in 3D (i.e. convert pixel (i, j) + depth (d) to 3D coordinates (x, y, z) per pixel).

[xuxw98]

Thank you for your reply!

One more question is about the coordinate system and scale of the pointmap / pointcloud. After I use camera intrinsics to convert depth image to pointcloud in camera coordinate system, should I do any other operation, like normalizing the pointcloud?

This is not about the format, but about which distribution of pointcloud (x,y,z, and maybe additional channels) SAM 3D is trained on.

[xuxw98]

I notice in the InferencePipelinePointMap method, even pointmap is not provided, a depth estimator will be adopted for depth estimation followed by intrinsics inferring. So I can replace the estimated depth and intrinsics with the real ones, is that right? Thank you!

[eternity123-null]

I notice in the InferencePipelinePointMap method, even pointmap is not provided, a depth estimator will be adopted for depth estimation followed by intrinsics inferring. So I can replace the estimated depth and intrinsics with the real ones, is that right? Thank you!

Hi @xuxw98, I also noticed that. So if I replace the estimated intrinsics with the real one and also input depth from my RGBD, does the "translation" and "rotation" in output denote the 6D pose of object? Thank you!

[alexsax]

@eternity123-null Yes the translation and rotation correspond to the 6D pose of the object, using Pytorch3d camera conventions

[alexsax]

@xuxw98

Sorry that I missed the main point of the issue you raised last week. To see how to provide your own pointmap/depth, you could look at the notebook @codepk37 contributed in #79