[Reproduction] Request for Official Evaluation Code to Reproduce MOKA’s Reported Results on LMMS-Eval Benchmarks (MMEpercep/MMBench/POPE/SEED-Bench)

[enfantsRichesDeprimes]

Dear authors,

First of all, thank you for your excellent work on MOKA! I truly appreciate the thoughtful design and impressive capabilities demonstrated in your paper—it has been a great source of inspiration for our own research.

I am currently trying to reproduce the evaluation results reported in your paper using the lmms-eval framework. Specifically, I have run evaluations on the following benchmarks: MMEpercep, MMBench, POPE, and SEED-Bench. However, despite carefully following standard procedures, the scores I obtained are consistently and significantly lower than those published in the paper.

Given the high quality of your work, I suspect that there may be important details—such as specific preprocessing steps, prompt templates, inference settings, or custom configurations within lmms-eval—that are not yet publicly available but are crucial for achieving the reported performance.

Therefore, I would be deeply grateful if you could kindly share the complete and official evaluation code used in your experiments.

Thank you very much for your time, effort, and contribution to the field. I sincerely hope this request is reasonable, and I greatly look forward to your guidance.

Warm regards.

[echo0409]

Thanks a lot for your interest!

We have updated the organized code, checkpoint and predicted results of the visual-text case!

Train

Stage 1 is the same as the audio-visual-text case. And the used visual projector is also the same: visual projector checkpoint.

Stage 2: SFT

sh MokA_VisualText/shell/train.sh

Download the used JSON of train data HERE. A small set of multiple-choice type instructions is integrated with the original LLaVA-Instruct-150K.

Inference

Use the last checkpoint. Take MMBench as an example, the command would be:

sh MokA_VisualText/eval_benchmarks/mmbench/mmbench.sh

Then, calculate the Acc/Score:

python MokA_VisualText/eval_benchmarks/mmbench/eval_mmbench.py

Results

Our predicted results are available in the merged.jsonl files located in the corresponding benchmark directories.

Dataset	Acc/Score
MME_perception	1105.51
MMBench	56.02
POPE	77.07
SEED-Bench	40.80

Our checkpoint

https://huggingface.co/yake0409/MokA_VisualText/tree/main

[enfantsRichesDeprimes]

Follow-up: Incomplete Evaluation Code Leads to Import Errors (modified_peft) When Reproducing MOKA Results

Dear MOKA Authors (@echo0409),

Thank you so much for your thoughtful and prompt reply to my previous inquiry—I truly appreciate the time and attention you’ve dedicated to supporting the community’s efforts to reproduce your work. I’m also very grateful for the release of the organized code, checkpoints, and evaluation results, which reflect a commendable commitment to transparency and reproducibility.

That said, I’ve run into a critical roadblock while trying to execute the provided evaluation pipeline—specifically, missing custom modules and unresolved dependencies that prevent the code from running at all.
Key Issue: Unresolvable Import Errors
When launching the inference or evaluation scripts (e.g., mmbench.sh), the process fails immediately with the following errors:

ImportError: cannot import name 'LoraConfig' from 'modified_peft'
ImportError: cannot import name 'PeftMixedModel' from 'modified_peft'

After thoroughly inspecting the entire released codebase—including both MokA_VisualText/ and MokA_AudioVisualText/—I found that:

1. The modified_peft module is not included anywhere in the repository;
2. There is no documentation or guidance on the structure of this module or how to install/build it;
3. Essential components such as custom PEFT/LoRA implementations—required for loading the checkpoint and initializing the model—are absent.
   Additional Context
   To rule out common setup issues, I have:
   Installed all standard dependencies exactly as specified in README_VT.md (including transformers==4.53.2, peft, torch, etc.);
   Verified that the checkpoint was correctly downloaded from your Hugging Face repository;
   Executed the evaluation commands precisely as instructed (e.g., sh mmbench.sh followed by python eval_mmbench.py).

Unfortunately, the absence of modified_peft halts the entire pipeline before it can begin—without it, the model cannot be instantiated, and no evaluation can proceed.
Request
To enable successful reproduction of your results, could you kindly:

1. Release the full modified_peft module, including the implementations of LoraConfig and PeftMixedModel used in your evaluation code?
2. Confirm whether any other components are missing (e.g., modified model classes, utility functions, or config files) that are necessary for end-to-end evaluation?

Thank you again for your continued support and openness. I sincerely look forward to your guidance in resolving these issues and accessing the complete, runnable codebase.

Warm regards.

[echo0409]

Sorry for the oversight. The folder was missing when downloading from the server. We have now added it here:
https://github.com/GeWu-Lab/MokA/tree/main/VisualText/modified_peft

Thanks a lot!