OI Eval and Training Logs

[liuhengyue]

Hi,

Fantastic work and thanks for releasing the codebase!

I noticed that your evaluation on Open Images seems different from other approaches. You are computing the mean per GT triplet, instead of per predicate class. You also skip computing for "unseen" triplet. I am not sure if these modifications would introduce big differences when comparing with other methods. Would you mind explain this?

I also would like to ask if you have the training logs that could share, especially on mAPs results for Open Images (object detection results per class, and predicate recall per class).

Thanks.

[JacobYuan7]

@liuhengyue
Hi, can you be more specific on "You are computing the mean per GT triplet, instead of per predicate class." and "You also skip computing for "unseen" triplet."? It is been a while since I use this codebase so I might forget some of the details.

In terms of the log, I will try to find one (I am a bit oocupied lately.).

[liuhengyue]

@liuhengyue Hi, can you be more specific on "You are computing the mean per GT triplet, instead of per predicate class." and "You also skip computing for "unseen" triplet."? It is been a while since I use this codebase so I might forget some of the details.

In terms of the log, I will try to find one (I am a bit oocupied lately.).

Thanks for your quick reply. I was refering to the code starting

RLIPv2/datasets/oi_sgg_eval.py

Line 234 in aa91a6b

for triplet in self.gt_triplets:

The ap is computed per GT triplet, whereas people compute per predicate class.

The skip for unssen triplet is related to the codes below

RLIPv2/datasets/oi_sgg_eval.py

Lines 372 to 373 in aa91a6b

	if triplet not in self.gt_triplets:
	continue

[JacobYuan7]

@liuhengyue
For the first query, according to my scope of knowledge, it is usually computed per triplet. Usually, performance per triplet will be lower than performance per predicate since we have much more classes.

For the second query, this is the default operation in mAP calculation for HOI detection. As I remember it, typical SGG mAP calculation has also done something similar to this. I assume that performance will degrade if we do not do this.

[JacobYuan7]

@liuhengyue
These are the training logs that you required. (For Swin-T, there are repetitve results due to multiple tries. Generally, they adhere to the reported results in the paper.)

RLIP_PDA_v2_OISGGtrain_SwinL_VGCOCOO365_RQL_LSE_RPL_20e_L1_20e.txt

RLIP_PDA_v2_OISGGtrain_SwinT_VGCOCOO365_RQL_LSE_RPL_20e_L1_20e.txt

RLIP_PDA_v2_OISGGtrain_R50_VGCOO365_RQL_LSE_RPL_20e_L1_20e.txt