Evaluation Code of T2I-CompBench

[yujinhanml]

Hi, thank you for your great work. I was wondering if you could consider releasing the evaluation code for T2I-CompBench, especially the parts related to: the evaluation metrics (e.g., UniDet, BLIP, etc.) and the exact models (e.g., Bert) for evaluation in Table 1. Thanks;)

[CaraJ7]

Hi @yujinhanml , we use the official evaluation code and settings in T2I-CompBench illustrated here.

[yujinhanml]

here

Thank you for your reply. I would like to confirm whether you used the understanding branch to enhance the prompts when evaluating Janus-Pro and T2I-R1. I directly used the original prompts for generation and found that the metrics were lower than those reported in the paper.

[CaraJ7]

We use the understanding branch during inference. The inference code is here.

[yujinhanml]

We use the understanding branch during inference. The inference code is here.

Thank you for your reply. May I further ask whether the understanding branch was also used when evaluating Janus-Pro-7B on the T2I-CompBench?

[CaraJ7]

We do not use the understanding branch when evaluating Janus-Pro-7B on T2IComp-Bench in our paper. Our experiments have revealed that incorporating the original understanding branch provides marginal improvements and, in some cases, even decreases performance:

	Color	Shape	Texture	Spatial	Non-Spatial	Complex
w/o und. branch	0.6359	0.3528	0.4926	0.2061	0.3085	0.3559
w/ und. branch	0.6451	0.3773	0.5262	0.2298	0.3049	0.3578

[yujinhanml]

We do not use the understanding branch when evaluating Janus-Pro-7B on T2IComp-Bench in our paper. Our experiments have revealed that incorporating the original understanding branch provides marginal improvements and, in some cases, even decreases performance:

Color Shape Texture Spatial Non-Spatial Complex
w/o und. branch 0.6359 0.3528 0.4926 0.2061 0.3085 0.3559
w/ und. branch 0.6451 0.3773 0.5262 0.2298 0.3049 0.3578

Thanks! I found that the performance numbers of Janus-Pro-7B on T2I-Bench seem to differ. Could you kindly provide more detailed information? The results I obtained are as follows:

Color	Shape	Texture	Spatial	Non-Spatial	Complex
 w/o und. branch	0.53	0.33	0.39	0.17	0.31	0.36

For Color, Shape, and Texture, I used BLIP-VQA; for Spatial, I used UniDet; for Non-Spatial, I used CLIP; and for Complex, I used 3-in-1 for evaluation. I would like to know whether the observed discrepancies are reasonable, or if I might have made some mistakes. Alternatively, do you have results from other papers regarding the performance of Janus-Pro-7B on T2I-CompBench? Thank you for your help.

[yujinhanml]

We do not use the understanding branch when evaluating Janus-Pro-7B on T2IComp-Bench in our paper. Our experiments have revealed that incorporating the original understanding branch provides marginal improvements and, in some cases, even decreases performance:

Color Shape Texture Spatial Non-Spatial Complex
w/o und. branch 0.6359 0.3528 0.4926 0.2061 0.3085 0.3559
w/ und. branch 0.6451 0.3773 0.5262 0.2298 0.3049 0.3578

By the way, for baseline show-o, the performance of show-o on T2I-bench I obtained is

Color	Shape	Texture	Spatial	Non-Spatial	Complex
 w/o und. branch	0.73	0.53	0.67	0.39	0.31	0.35

which is quiet different in the paper

Color	Shape	Texture	Spatial	Non-Spatial	Complex
 w/o und. branch	0.56 0.41 0.46 0.20 0.30 0.29

I am concerned that I may have caused this error on my side. Thank you again for your help.

[yujinhanml]

Please take a look at this work, which reports the performance of Janus-Pro-7B on T2I-CompBench (Table 3), for your reference: https://arxiv.org/pdf/2505.05470.

[CaraJ7]

For T2IComp-Bench, we follow the official generation setting, where we generate 10 images for each prompt with a random seed set as 42:

from pytorch_lightning import seed_everything

for prompt in prompt_list:
    # preparing the input prompt similar to https://github.com/deepseek-ai/Janus/blob/main/generation_inference.py
    ...


    # set random seed for each prompt
    seed_everything(42)

    # generate 10 images for 1 prompt
    generate(
        vl_gpt,
        vl_chat_processor,
        prompt,
        prompt_text,
        num_generation=10,
        conversation=conversation
    )

[yujinhanml]

For T2IComp-Bench, we follow the official generation setting, where we generate 10 images for each prompt with a random seed set as 42:

from pytorch_lightning import seed_everything

for prompt in prompt_list:
# preparing the input prompt similar to https://github.com/deepseek-ai/Janus/blob/main/generation_inference.py
...

# set random seed for each prompt
seed_everything(42)

# generate 10 images for 1 prompt
generate(
    vl_gpt,
    vl_chat_processor,
    prompt,
    prompt_text,
    num_generation=10,
    conversation=conversation
)

Hi, thanks for your reply. May I ask more details about the evaluation? For example, the metric for each subtask ? It’s quite weird the performance of Janus and showo are so different when comparing with my numbers and the results (tab 3) in this paper https://arxiv.org/pdf/2505.05470.

[CaraJ7]

Hi, we use exactly the same metric as yours:

For Color, Shape, and Texture, I used BLIP-VQA; for Spatial, I used UniDet; for Non-Spatial, I used CLIP; and for Complex, I used 3-in-1 for evaluation