Is there a reason why if you specify 500 images per class, you get slightly more than 500

[CMCDragonkai]

Running

python ./downloader.py -data_root ./tmp -number_of_classes 1000 -images_per_class 500

This in classes with 507 images, 505 images, 507 images... etc. What's the reason for this slight more than 500 images?

Does this have something to do with the concurrent downloading, and the fact that some URLs no longer work?

[CMCDragonkai]

Also I read https://mf1024.github.io/2019/06/09/how-to-scrape-the-imagenet/ which is great research on this issue. Imagenet is not really a reliable source of data.

However nowhere does it indicate why there slightly more images per class than requested.

[huntkao]

Does anybody know ? thanks.

[ogkalu2]

@huntkao @CMCDragonkai
It's the multiprocessing workers. It's fucking with the code. Set it to 1 and it'll download only as many as you specify