Artifacts in Odom/SLAM trajectory on stereo camera

[raghavauppuluri13]

Thanks for putting out cuVSLAM! I've been working on adding cuVSLAM support for a low-cost USB-C camera and have been running into some issues with the visual odometry/slam quality, I see consistent 'spikes' in the trajectory and some off axis drift (should be a planar U trajectory). I have a google drive folder with a dataset/code and camera calibrations that reproduce the behavior I am seeing along with some notes in the README.md. I am running on CUDA 12.6 on the Jetson Orin Nano.

Some notes about the stereo camera/setup: does not have hardware timestamping, extreme fisheye and ~90-100ms of latency from hardware frame capture. The stereo pair images are synchronized before usb transport. Unfortunately, the camera is not capable of other combinations with higher resolution at higher frame rates than 240x480 image pairs at 30hz and 480x840.

Any suggestions?

https://drive.google.com/drive/folders/1DjuwbDikYrXLWq9DcJidbcy6-Nnt5NqE?usp=sharing

[raghavauppuluri13]

update: planar_constraints=True helps, but is a hacky solution, would like to know why drift occurs without it

[zwdoescode]

Hi @raghavauppuluri13! I found a few issues so far:

1. By looking at your sensor calibration file and use it to undistort images, I suspect that the calibration is not correct. Please double check your calibration (intrisincs) and recalibrate if necessary
2. Part of the robot body is in the FOV of the camera and features are being extracted and used by PyCuVSLAM, this would downgrade the performance of tracking. You can apply an image mask to filter out the robot body. Please check out the API doc for Track.track
3. There are too many keyframes - tracks die without visible reason. Try decrease resolution to VGA and enable denoise filtering. Or try to inspect raw original images frame by frame to understand reason. Noise? Image compression? Prediction?

Here are some tips for troubleshooting (we will make a formal version later):

1. Check calibration
2. Check camera sync
3. Check pure mono left / right tracking
4. If mono mode provides good trajectory but there are jumps in stereo, that means you need to recalibrate left_from_right matrix

[Somanshu-Jha]

Hi, I would like to work on this issue.
Please assign it to me.

[zwdoescode]

@Somanshu-Jha Sure. Please feel free to create a PR. We will have engineers to evaluate it internally. Thanks for your interests in contributing this repo!

[Somanshu-Jha]

Thanks for the reply, I'll work on this repo and provide the best as I can