Training time

[xiaoyudanaa]

Excellent work! I have a question: you mentioned in the paper that the obman dataset ran for 60 hours on 4x3090 with a batch_size of 256. However, when I followed the training steps you outlined in the readme and ran your code on 2x3090, I found it to be quite slow: 1 epoch took 1 hour with my batch_size set to 128 (due to memory constraints). Do you know why this might be?

[zerchen]

Thanks for your kind words. According to my experience, I think the main bottleneck is the data loading issue. Since SDF files contain so many .npz files, it would be expensive to decode them on-the-fly in the training time. A better choice will be use LMDB (also provided as an option in this repo) to accelerate the data fetching. Sometimes, I also observe that the first epoch of my training is very slow (probably the machine was creating caches), but it will be fine in later epochs.

[xiaoyudanaa]

Thank you for your response. I noticed that in the config.py of your project, the default setting is cfg.use_lmdb = False. Should I simply change it to True? (Prior to this, I have already preprocessed the dataset)

[zerchen]

Yes, turning it to True should be fine.

[xiaoyudanaa]

I'd like to make a correction. Your training command "bash dist_train.sh 4 1234 -e ../playground/hsdf_osdf_1net/experiments/obman_resnet18_hnerf3_onerf3.yaml --gpu 0-3 use_lmdb True" seems to have already set lmdb to True. I have been using lmdb previously.

[zerchen]

ok, does the training speed remain slow after the 1st epoch?

[xiaoyudanaa]

Thank you for your concern. After realizing the project was running too slowly last night, I terminated the process. Now, the server is occupied by someone else. I'll update you once I've conducted the tests.

[xiaoyudanaa]

ok, does the training speed remain slow after the 1st epoch?
I apologize for the interruption. May I have your contact information? I have a few questions I'd like to discuss with you. My WeChat ID is: Weafree. I would greatly appreciate it.