scale max_concurrent_uni_streams with BDP

[alexpyattaev]

Problem

Streamer ignores RTT in all its calculations. This means that connections with high latency are heavily rate limited, much more than the stake amount would suggest. This happens before the actual stake-based stream throttling even has a change to kick in, and is thus very counterintuitive to the client.

A more principled version of #7954

Some preliminary concepts:

• Sender can not have more than max_concurrent_streams worth of open streams at any time
• These together limit how many TXs can be “in flight” between client and server and not yet ACKd. Currently, these limits are computed based on stake. We need to compute them based on stake and RTT to the client (as longer RTT means you need to have more things on the wire before you see an ACK).
• Beyond all the limits mentioned above, there is an overall stream limit imposed by the code in stream_throttle.rs. That is out of scope of this PR.

This mechanism is not intended as the actual rate limiter for complaint clients, just as a limit on network buffers and such like. This is designed to allocate more bandwidth than what you'd get today.

Summary of Changes

• Assume that in the original code the RTT was expected to be ~50ms
• Scale RX window and max_streams with RTT if it is above 50ms

Whether 50ms is the right latency to pick is a good question. 50 ms was chosen as a compromise between allowing more TPS and keeping changes small.

Impact

Before:

After:

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.6%. Comparing base (bd0b3d3) to head (10d60db).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #8948   +/-   ##
=======================================
  Coverage    82.6%    82.6%           
=======================================
  Files         844      844           
  Lines      316630   316633    +3     
=======================================
+ Hits       261595   261610   +15     
+ Misses      55035    55023   -12     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[alexpyattaev]

Additional testing against testnet nodes on public internet. Test run duration: 10 seconds, repeated 3 times, using 2 staked identities with 2M testnet sol each:

With scaling
	RTT, ms	#1	#2	#3	#4	#5	#6	Mean	Std
SLC-Dallas	30	9793	9784	9763	9811	9790	9788	9788	15
London-Dallas	100	7543	8537	8471	8488	8496	8510	8341	391
Frankfurt-Dallas	110	8353	8303	8337	8321	8350	8340	8334	19
Tokyo-Dallas	130	8163	7955	8143	7947	8136	8100	8074	97
Singapore-Dallas	220	5961	7500	6916	5796	7429	6054	6609	768
Without scaling (Master)
	RTT, ms	#1	#2	#3	#4	#5	#6	Mean	Std
SLC-Dallas	30	9793	9784	9771	9790	9783	9758	9780	13
London-Dallas	100	3940	4625	3600	3614	4630	4475	4147	489
Frankfurt-Dallas	110	4134	4094	4133	4136	4041	4131	4112	38
Tokyo-Dallas	130	3043	3054	3426	3094	3411	3053	3180	185
Singapore-Dallas	220	1538	1465	2065	1543	2079	1524	1702	288

As intended, on longer links (Tokyo, Singapore) the gains are massive and result in almost same TPS as within the US. London-Dallas link is particularly bad quality (high packet loss) but is consistent with the rest of the results (but with much higher standard deviation).

[t-nelson]

are the numbers in the chart the measured open streams or the calculated stream limit?

[gregcusack]

are the numbers in the chart the measured open streams or the calculated stream limit?

the numbers are TPS (besides the RTT in ms of course)

[alexpyattaev]

are the numbers in the chart the measured open streams or the calculated stream limit?

The numbers in the table are measured TPS.

[gregcusack]

thank you for getting these numbers (both sim and on real network)! just the question on attack vector here (think it was brought up on backport call as well).

[gregcusack]

Where does this number come from? great that we have a cap here! is it possible to spoof your rtt when you initially setup your connection to get more streams? And then you get more access to the pipe? For example, if your RTT is 20ms but you spoof your connection to be 350ms, how much more TPS are you going to be able to get?

maybe this is not a big issue but just want to make sure.

[alexpyattaev]

320 ms is an RTT from Australia to Portugal, which is the largest RTT I could find on this planet. 350 is that + a bit of margin.

You can of course spoof RTT and get more concurrent streams, but you would get throttled by SWQOS to whatever your proper quota is anyway. So faking RTT is not going to bring you real TPS increases beyond what you are supposed to get based on your stake.

[gregcusack]

got it! thanks for the explanation!

[stablebits]

With "no throttling below threshold" in place there will be no throttling in SWQOS unless the overall load is high though. It should be possible to address RTT spoofing, if really needed.

[alexpyattaev]

If someone will be trying to abuse this, all they will achieve is force system back into SWQOS mode, which they can do without BDP scaling easily enough (as we did during tests of #9580)

sorry approved by accident. just want some clarity on attack vector here first. otherwise looks good

[gregcusack]

lgtm! thanks for putting this together! we probably want another review from @stablebits and/or @alessandrod as well

[mergify]

Backports to the beta branch are to be avoided unless absolutely necessary for fixing bugs, security issues, and perf regressions. Changes intended for backport should be structured such that a minimum effective diff can be committed separately from any refactoring, plumbing, cleanup, etc that are not strictly necessary to achieve the goal. Any of the latter should go only into master and ride the normal stabilization schedule. Exceptions include CI/metrics changes, CLI improvements and documentation updates on a case by case basis.