Transformers Version

[mdivyanshu97]

Dear Authors,

Amazing work. I was trying to reproduce the results but was getting errors in different open-source LLM's due to the transformers version. Can you kindly let me know the exact transformers version you used to run all of them?

The current setup file just says transformers>=4.39.2 but when using versions like 4.42 some LLMs are not present on hugging face.

[videommmu]

Hi @mdivyanshu97 ,

Thank you for your interest in our work!

The transformers requirement in our setup reflects the minimum version needed to support the default models used in earlier versions of lmms-eval. So the exact version depends on the specific model you're evaluating. More recent LLMs require more recent releases of transformers like 4.51.0+ to load correctly.

Please feel free to try 4.51.0 as a starting point for evaluation.

All the best with your work!

Best regards,