[GAD] Question regarding Reward Score fluctuations and downstream evaluation

[ZhangChi315]

Thank you so much for sharing your work on GAD! It’s an impressive project, and we’ve been very interested in exploring its implementation.

While reproducing the results, we encountered a few points where we’d love to get your expert guidance:

1. Observations on Reward Score Behavior In our experiments, we noticed that the reward scores exhibit quite significant fluctuations and tend to hover around 0. This seems a bit different from the typical upward trend and eventual stabilization we often see in other RL frameworks.

We were wondering if this behavior is expected for GAD?

From your experience, what are the best indicators or metrics to look for to determine if the model has reached sufficient convergence?

2. Downstream Benchmarks We are also curious if you have had the chance to test GAD on more common downstream evaluation sets (such as IFEval). If so, we would be very interested to hear your thoughts on those results!

Thank you again for your contribution to the community. We look forward to hearing from you whenever you have a moment.

[YTianZHU]

1. Reward‑score behavior
   The oscillation around 0 is expected for GAD. To gauge convergence, use the automatic‑evaluation (AE) score saved with each checkpoint and pick the checkpoint where AE plateaus (or peaks). That AE score is the most reliable indicator of sufficient convergence.

2. Downstream benchmarks
   On common tasks such as IFEval (and others we tested), GAD’s performance is on par with the original model & SeqKD baseline—differences are minor.