[GAD] Plans to release the test data?

[luferw]

Do you have plans to release the test datasets used in the paper (500 samples from LMSYS-Chat-1M-Clean, 500 from Dolly, 252 from SelfInst, 80 from Vicuna)? This would greatly help with reproducibility.If full release is not possible, could you share the sampling/filtering methodology to allow reproduction?

[luferw]

!I noticed that the test sets for Dolly, SelfInst, and Vicuna are already available in the MiniLLM Hugging Face repository. However, I couldn’t find the 500 test samples from LMSYS-Chat-1M-Clean mentioned in the paper.

Are there any plans to release this part of the test data? Releasing it would greatly help with reproducibility and further research.

[YTianZHU]

@luferw Hi, test data is just split from the provided huggingface LMSYS-Chat-1M-Clean. You can just randomly split by yourself. Because we use GPT-4o as a judge in the evaluation, it is not quite possible to get exactly same evaluation score as in paper, considering maybe you use other version/model to evaluate. So just randomly take 500 from the provided dataset is fine.

[luferw]

@YTianZHU Thank you for the clarification! I will proceed with sampling from the dataset for testing.

To better reproduce the experiments, I have two follow-up questions regarding the checkpoint selection strategy mentioned in the paper ("selecting the checkpoint with the highest GPT-4o score every 50 steps"):

1. Optimal Step Range:Across different experimental setups, around which step range was the final optimal checkpoint typically selected? Did the GPT-4o score generally increase monotonically, or did it fluctuate (e.g., dip and then rise) during training?
2. Metric Correlation: In your experiments, were the trends of ROUGE-L and GPT-4o scores generally consistent? For instance, in my own adversarial training runs, I observed a sharp drop in ROUGE-L around 1500 steps followed by a recovery. Did GPT-4o scores ever show similar decoupled fluctuations in your setup?

Understanding these points would be very helpful for assessing training progress and ensuring reproducibility. Thank you again!

[YTianZHU]

@luferw

1. The GPT-4o score fluctuates, because it is RL process and there is large variance. The overall score trend of GAD increases and then saturate with fluctuation in the middle.
2. ROUGE-L basically has nothing to do with GPT-4o scores. As in the ReadMe:

We observe that higher ROUGE-L scores do not necessarily correspond to better performance in either automatic or human evaluations. Consequently, ROUGE-L is used solely as a training diagnostic to verify that optimization is proceeding normally.

GPT-4o score align with human evaluation more than ROUGE-L.

[luferw]

@YTianZHU Thank you so much for your timely response. I would also like to ask whether the scores reported in Table 2 represent the proportion of instances where the model’s response scored higher than GPT-4o’s response, based on GPT-4o's evaluation. Does this include cases where the scores are equal, or does it represent the average score given by GPT-4o to the model’s responses?

What seems strange is that for my untrained model Qwen2.5-7B-Instruct, the average scores across all four datasets are above 7, while the proportion of instances where the model’s score exceeds GPT-4o’s score is almost always below 10%. In contrast, the proportions reported in the paper are above 45%.

Here is an example of a response from my untrained model:

{
  "input": "system\nYou are a helpful assistant.\nuser\nBelow is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nIdentify which instrument is string or woodwind: Panduri, Zurna\n\n### Response:\n\nassistant\n",
  "output": "The Panduri is a string instrument, while the Zurna is a woodwind instrument.",
  "teacher_output": "- **Panduri**: String instrument  \n- **Zurna**: Woodwind instrument",
  "_compare_key": "21-df5213a4feefc86e5ae8b7cb886abc42",
  "gpt_compare_response": "9 10  \nAssistant 2's response is slightly more organized and clear because it separates the two instruments with individual bullet points, making it visually easier to read and interpret. Both responses are accurate, correct, and relevant, but Assistant 2 provides a clearer presentation format, which enhances its helpfulness and readability. Hence, it deserves a slightly higher score of 10 compared to Assistant 1's 9.",
  "score_1": 9,
  "score_2": 10,
  "winner": 2
}

and the compare prompt is

system\nYou are a helpful assistant.\nuser\nBelow is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nIdentify which instrument is string or woodwind: Panduri, Zurna\n\n### Response:\n
### Assistant 1 Response:\n{assistant1}\n
### Assistant 2 Response:\n{assistant2}\n\n"
We would like to request your feedback on the performance of two AI assistants in response\......the prompt in Figure 8: GPT-4o evaluation prompt.....

Could this discrepancy be due to differences in the prompt? The Dolly, SelfInst, and Vicuna datasets are public datasets, and the model has not been fine-tuned. I have not modified the code of VERL and directly used the provided script command for inference:

CUDA_VISIBLE_DEVICES=4,5 bash scripts/generate/generate.sh --model /data/models/Qwen2.5-7B-Instruct --exp_name Qwen2.5-7B-Instruct-untrained --val_data ${val_data} --ckpt_start 0 --ckpt_end 0 --ckpt_step 1 --nnodes 1 --ngpus 2 &

Thank you once again for your patience and invaluable assistance!

[YTianZHU]

@luferw
Hi, sorry that we didn't present the score calculation clearly. GPT-4o score is defined as
$\frac{StudentScore}{(StudentScore + RefScore)}$
So for your example it's $\frac{9}{9+10}$

We will update this more clearly in the paper. Thanks for pointing out!

[luferw]

@luferw Hi, sorry that we didn't present the score calculation clearly. GPT-4o score is defined as S t u d e n t S c o r e ( S t u d e n t S c o r e + R e f S c o r e ) So for your example it's 9 9 + 10

We will update this more clearly in the paper. Thanks for pointing out!

I got it!Thank you so much!