Inference using single_turn_eval.py

[nsrishankar]

Hi,

Thank you for releasing the code base and the weights. I'm having an issue when running the single_turn_eval.py script using torchrun --master_port=1112 --nproc_per_node=2 single_turn_eval.py , the default arguments, and the weights obtained from Huggingface https://huggingface.co/FanqingM/ChartAssistant/tree/main. I get a dimension mismatch error below
RuntimeError: The size of tensor a (1025) must match the size of tensor b (257) at non-singleton dimension 1 . Please let me know if there is a solution.

Thanks!

[lwl1751]

I encountered the same issue. Have you managed to solve it? When I changed the image size from 448 to 224, it was able to run, but the output results were very poor. I haven't found a solution yet.

[nsrishankar]

I have not and ended up dropping this approach.

[lwl1751]

I have successfully run the code. In the single_turn_eval.py script, the --llama_type parameter defaults to 'llama_qformerv2'. I changed it to 'llama_ens5' to be consistent with the setting in single_turn_eval_multitask.py. Under this configuration, the image size can be set to 448 without needing to downscale to 224.

However, the test performance I obtained is significantly worse than the results reported in the paper. The example output from Figure 9 in the paper is as follows:

entity 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0
MiniGPT-5 37.0 32.1 31.2 30.3 30.3 30.3 30.6 31.2
Stable Diffusion 2 30.4 27.3 26.4 26.6 26.8 27.0 27.5 27.9

The output I obtained is:

Stable Diffusion Version MiniGPT-5 Stable Diffusion 2
2.5 37.2 30.5
3.0 32.3 27.2
3.5 31.5 26.4
4.0 30.4 26.2
4.5 30.4 26.1
5.0 30.3 25.9
5.5 30.3 25.7
6.0 30.3 25.6
6.5 30.3 25.4
7.0 30.3 25.2
7.5 30.3 25.0
8.0 30.3 24.8
8.5 30.3 24.7
9.0 30.3 24.5
9.5 30.3 24.4
10.0 30.3 24.3

This is what I’ve found so far.