Couldn't reproduce ImageNet accuracy 72.19% with student pretrained=False

[yoshitomo-matsubara]

Hello @ADLab-AutoDrive

Thank you for open-sourcing your work!
I wanted to reproduce the 72.19% in Table 2, which I assume corresponds to this configuration file

I found that the student model did not train from scratch as pretrained: True

Because it was not described in the paper and doesn't look like a fair comparison (other baselines in Table 2 trained student model from scratch), I thought it was a mistake and reran the experiment with pretrained: False, but the resulting accuracy was 68.34%, which is far from the reported result.

Could you clarify configs used in the paper?

[yoshitomo-matsubara]

Hi @ADLab-AutoDrive

I got a number using pretrained: True for the student model as used this configuration file

• student pretrained: True -> Top-1 accuracy 71.9660 %
• student pretrained: False -> Top-1 accuracy 68.3440 %

Based on these numbers, the presentation of the improved results for ImageNet dataset reported in the paper (72.19% in Table 2) looks unfair as other baseline methods presumably trained models from scratch (i.e., training the student model with pretrained: False for 90 - 100 epochs) while the config in this repository implies ICKD-C used 190 < training epochs (90 epochs for pretrained: True in torchvision used for model initialization and 100 epochs with torchdistill you used for training).

Could you clarify the experimental setting actually used for ImageNet dataset? I couldn't find such descriptions in the paper (e.g., Section 4.2)