Unable to reproduce results

[KwanWaiChung]

Hi, thank you for releasing the code and paper — I really appreciate the work!

I'm having difficulty reproducing the results reported in the paper and would appreciate any guidance you could provide.

I followed the instructions in README.md exactly, with two modifications due to hardware constraints:

• Reduced number of GPUs (please specify how many you used if relevant)
• Switched FAISS to CPU due to GPU memory limitations

I disabled validation during training. Evaluation results were obtained using the following script:

PYTHONUNBUFFERED=1 python -m verl.trainer.main_ppo \
    --config-name='search_multiturn_grpo' \
    data.train_files="$TRAIN_DATA" \
    data.val_files="$VAL_DATA" \
    data.train_batch_size=256 \
    data.val_batch_size=256 \
    actor_rollout_ref.model.path=${model} \
    actor_rollout_ref.model.enable_gradient_checkpointing=true \
    actor_rollout_ref.model.use_remove_padding=True \
    actor_rollout_ref.rollout.name=sglang \
    actor_rollout_ref.rollout.tensor_model_parallel_size=${tp} \
    actor_rollout_ref.rollout.gpu_memory_utilization=0.5 \
    actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=2 \
    actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=32 \
    actor_rollout_ref.rollout.n=1 \
    actor_rollout_ref.rollout.multi_turn.tool_config_path="$TOOL_CONFIG" \
    actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=32 \
    actor_rollout_ref.ref.fsdp_config.param_offload=True \
    algorithm.adv_estimator=grpo \
    trainer.logger='[]' \
    trainer.val_only=true \
    trainer.val_before_train=true \
    trainer.n_gpus_per_node=${gpus} \
    trainer.nnodes=1 \
    trainer.validation_data_dir=${ROLLOUT_DIR} \
    "$@"

My results of Qwen 3b iter2 and 7b iter 1 are quite different from the numbers reported in the paper.

Model	NQ	TriviaQA	PopQA	HotpotQA	2WikiMQA	Musique	Bamboogle	Avg
Qwen-2.5-3b-iter1	37.26	54.42	37.82	28.87	26.46	10.47	28.80	32.01
Qwen-2.5-3b-iter2	37.20	54.31	38.62	26.62	19.01	6.79	16.00	28.36

Model	NQ	TriviaQA	PopQA	HotpotQA	2WikiMQA	Musique	Bamboogle	Avg
Qwen-2.5-7b-iter1	40.14	57.09	40.50	28.25	19.95	6.79	16.80	29.93

Full training logs are available here: https://wandb.ai/cyruskwan/dr-zero-public. I would greatly appreciate any guidance or suggestions that could help me reproduce the reported results. Thank you!

[Yueeeeeeee]

Hi, thanks for you interest in our work! Please ensure that your evaluation script loads the trained model parameters with the same configuration used for training. Regarding your 3B model, the strong multi-hop results in iteration 1 followed by a sudden drop in iteration 2 suggest that the quality of the multi-hop questions is degrading in the second iteration. I recommend inspecting the generated questions to verify this, you might also consider increasing the ratio of multi-hop questions in the dataset to improve multi-hop performance.

For the 7B model, we also observed more stability issues caused by inconsistent tokenization in multi-turn RL in our experiments, this likely explains why your 7B model performs worse than the 3B model. To prevent the training collapse seen in your logs, please apply a KL divergence penalty to stabilize your GRPO training. I also recommend checking verl-project/verl#3720 or upgrading your sglang and verl implementations if you still encounter this issue.

For reference, we use 8 A100/H100 GPUs for training and perform evaluation after 50 steps using the default validation loop built in verl.

[KwanWaiChung]

Thanks for the reply! For evaluation, I used the checkpoints after training for 50 steps and made sure the evaluation configurations were identical (greedy decoding, using 'search_multiturn_grpo', etc.). While the 7B model did experience training collapse in later stages, I'm surprised that the performance at step 50 is still underperforming, which seems to occur well before the collapse.

Thanks for pointing out the tokenisation issue! I'll try retraining with the default validation loop at step 50 to see if there are any differences. I'll also check the generated questions from the 3B models. I'll check back in a few days, and if the issue persists, I can upload the model so we can investigate further.

Thanks again for taking the time to help out—really appreciate it!