Reproducing CODI score

[JuseonDo]

Congratulations!
Finally SIM-CoT is accepted at ICLR!!

I have some issues on reproducing CODI score. I was able to get 42% on the validation set(GSM8K-Aug) as described in your paper but I can't reach 43%(reported score) on the test set(GSM8K).

How about you? Were you able to reproduce the CODI score on GSM8K dataset?

I used official code but I wasn't able to get 43% accuracy on test set. Please let me know!

[Wiselnn570]

Hey, thanks.

Me too — I can only get around 42% as well, which is consistent with the results reported in Table 1 (last row) of the paper (https://arxiv.org/pdf/2509.20317).

[JuseonDo]

@Wiselnn570 Thank you for replying.

I am so sorry for bothering you. I think the question wasn't delivered well.

I was wondering that how was the CODI results on the GSM8K(not GSM8K-Aug) which is used as test dataset in CODI paper.

In your experiments, the in-domain results are based on GSM8K-Aug test split which is used for validation in training step. right? (As described on your code, I am thinking you reported validation results on the main tables)

On the validation, I was also able to get 42% with CODI setting like as your paper. But the accuracy of the best checkpoint wasn't able to get 43% not even 42%. It's around 35%~37%.

So I want to know that you were also not able to get the reported score(around 43%) on the GSM8K.

Lastly, congratulations again! And thank you.

Best,
Juseon Do

[Wiselnn570]

I didn’t quite understand what you meant. You obtained a result of 42, but it isn’t from your best checkpoint.

[JuseonDo]

@Wiselnn570 Sorry for my ambiguous writing.

In training step, I was able to get 42% in validation but not test step.
CODI paper didn't report validation results but test results on GSM8K test split.

Based on your code, I found that the reported "GSM8K-Aug In-domain" results are not test results but validation results.
Since the GSM8K-Aug test split was used for the evaluation during the training step, the results are not test results but validation results.

During training step, I was also able to get 42% in validation. But in test step(I used GSM8K test split), the best checkpoint which reached 42% in validation step was scored only about 35~37% accuracy.

Not only me but also other people are suffering from reproducing CODI test accuracy(on GSM8K test split).("zhenyi4/codi#12" at this issue he also got 37% accuracy.)

So, I just wanted to know how much accuracy did you get on the test.
Also, I wanted to ask whether the reason you reported validation accuracy instead of test results was due to the CODI reproduction issues.

[Wiselnn570]

These results are from the GSM8K test set

[JuseonDo]

@Wiselnn570 Thank you for your replying.

Aha I misundertood your results. I am so sorry.

Since, in your paper Section 4.1, the dataset details are described as following, I thought you reported the results on the GSM8K-Aug test split:
"Evaluation Benchmarks. We report results on the GSM8k-Aug test set (Cobbe et al., 2021), ..."

Best Regards,
Juseon Do

[JuseonDo]

@Wiselnn570 At first, I realized that the mismatch between my validation and test results was caused by a simple bug in my code. While fixing it, I noticed something else that I wanted to ask about.

It seems that the test split of gsm8k-aug is exactly the same as the test split of the original GSM8K dataset. If that’s the case, using GSM8K test performance for early stopping or model selection would effectively be test-set tuning, so having a separate validation set seems important.

However, I couldn’t find details about the validation setup in the paper. Could you clarify which dataset or split was used for validation, or how validation was handled in your experiments? It’s also possible that I missed something, so I’d appreciate any pointers.

Best Regards,
Juseon Do

[Wiselnn570]

You can find validation set here.(https://huggingface.co/datasets/whynlp/gsm8k-aug)