training on vcr task

[huangsiyong]

hi, i notice that when finetuning ssp model on vcr task, the performance drop a lot at each 5000 steps in the first epoch.
before finuetuning, the result for Q2A and QA2R are both more than 74%
step 5000: 67% and 66%
step 10000: 64.7% and 64.9%
step 15000: 61.3% and 60.9%
step 20000: 60.8% and 57.9%
and the other results have no been gained. still training.

is it true? is the model trained in a right way? have you notice this phenomenon when you finetune it?
of course, i have no V100 so i train the model in 4 2080ti. with the limitation of memory, i set batch size=1, test_batch_size=4, gradient_accumulation_steps=8. the other config is the same as your vcr.yaml.

looking for you help, thanks!

[qyc-98]

Hi, I fine-tuned it in a larger batch size,(e,g in 32 V100 or 64 A100). You can try a larger gradient_accumulation_steps.

[huangsiyong]

ok, i will try it. thanks

[huangsiyong]

hi, on this weekend i try to run in 16 2080ti with batch size=1 and gradient_accumulation_steps=32.
acc after epoch 0 is 66.7%, epoch 1 is 66.3%.
As the training time is so long and there is no increase after the first two epochs, i want to know whether i should still run, in other words, whether there is something wrong.
or can we frozen some parameters of the model to maintain its performance and speedup the finetune training?

[huangsiyong]

i evaluate the pevl_vcr_finetune.pth and pevl_vcr_ssp.pth, which are provided, on the val set.
Acc of two checkpoints are 74.1/74.8 and 74.2/74.8.
Is it true? There is no improvement and a gap if comparing it with the result in the paper.