server based multi-turn training load balance

[jQizhang]

Hi, I have a question.

In verl/experimental/agent_loop/agent_loop.py, a heap data structure is used to implement load balancing. When a request is sent to a server, the corresponding server's request_num is +1. However, when the request is completed, the server's request_num is not -1.
Wouldn't this prevent it from reflecting the real-time number of requests being processed?

[wuxibin89]

Yes, the load balance strategy is quit naive for now. In first turn, it choose a server with least-request, and in follow turns, it pin request on that server for better kv cache use. So it's equivalent to round-robin load balance, we may need more kv cache utilization aware load balance strategy.

[chenhaiq]

We have implemented a load balance strategy in a batch, but found that it is not helpful to improve performance, because it have to reduce the parallel running samples in order to have the chance to place some samples to relatively idle gpus. Although work load is balanced in this way, the performance may become poor because the parallel running samples is lower.

The better approach is to load balance samples between batches, there is a PR working on it: #2200

[jQizhang]

Great Work! Thanks. I have a question:

"this strategy works since we basically select the shortest generation samples to fill this batch and postpone those long tail samples "

If the model mainly learns from the short samples, will this cause the response length to increase more slowly during training? Did you observe a similar phenomenon?