Benchmark Transfer PoV Size with Increasing Numbers of Users

[shawntabrizi]

This is a new #[extra] benchmark in the balances pallet which generates accounts with a balance, and then executes a worst-case-scenario balance transfer.

We can use this to model the growth in PoV size as the number of users increase.

Compiling with:

cargo build --release --features runtime-benchmarks

Then running:

./target/release/substrate benchmark --chain=dev --record-proof --pallet=pallet_balances --extrinsic=transfer_increasing_users --raw --repeat=1 --high=300000 --steps 300

We get the results:

Pallet: "pallet_balances", Extrinsic: "transfer_increasing_users", Lowest values: [], Highest values: [300000], Steps: [300], Repeat: 1
u,extrinsic_time_ns,storage_root_time_ns,reads,repeat_reads,writes,repeat_writes,proof_size_bytes
0,64000,11000,1,18,1,4,1806
1000,58000,4000,1,18,1,4,3302
2000,79000,5000,1,18,1,4,3410
3000,65000,4000,1,18,1,4,3509
4000,67000,3000,1,18,1,4,3707
5000,69000,4000,1,18,1,4,3839
6000,66000,4000,1,18,1,4,3938
7000,69000,3000,1,18,1,4,3971
8000,69000,4000,1,18,1,4,4004
9000,215000,4000,1,18,1,4,4103
10000,73000,4000,1,18,1,4,4173
11000,91000,5000,1,18,1,4,4173
12000,83000,4000,1,18,1,4,4173
13000,89000,4000,1,18,1,4,4206
14000,92000,5000,1,18,1,4,4239
15000,101000,5000,1,18,1,4,4272
16000,99000,5000,1,18,1,4,4272
17000,99000,5000,1,18,1,4,4272
18000,104000,5000,1,18,1,4,4272
19000,101000,4000,1,18,1,4,4305
20000,156000,5000,1,18,1,4,4338
21000,102000,5000,1,18,1,4,4404
22000,100000,5000,1,18,1,4,4404
23000,112000,5000,1,18,1,4,4404
24000,111000,5000,1,18,1,4,4404
25000,105000,5000,1,18,1,4,4404
26000,111000,5000,1,18,1,4,4470
27000,115000,6000,1,18,1,4,4536
28000,105000,5000,1,18,1,4,4536
29000,114000,5000,1,18,1,4,4536
30000,125000,5000,1,18,1,4,4536
31000,138000,6000,1,18,1,4,4536
32000,130000,7000,1,18,1,4,4569
33000,124000,6000,1,18,1,4,4602
34000,159000,6000,1,18,1,4,4602
35000,110000,5000,1,18,1,4,4635
36000,113000,6000,1,18,1,4,4800
37000,112000,5000,1,18,1,4,4800
38000,132000,6000,1,18,1,4,4800
39000,123000,5000,1,18,1,4,4800
40000,197000,6000,1,18,1,4,4833
41000,126000,6000,1,18,1,4,4833
42000,129000,6000,1,18,1,4,4866
43000,120000,6000,1,18,1,4,4866
44000,119000,7000,1,18,1,4,4866
45000,129000,6000,1,18,1,4,4866
46000,138000,9000,1,18,1,4,4866
47000,137000,7000,1,18,1,4,4866
48000,131000,6000,1,18,1,4,4866
49000,136000,7000,1,18,1,4,4866
50000,117000,7000,1,18,1,4,4899
51000,129000,6000,1,18,1,4,4899
52000,172000,6000,1,18,1,4,4932
53000,132000,6000,1,18,1,4,4932
54000,124000,6000,1,18,1,4,4932
55000,134000,5000,1,18,1,4,4932
56000,130000,7000,1,18,1,4,4932
57000,130000,5000,1,18,1,4,4932
58000,134000,8000,1,18,1,4,4932
59000,115000,5000,1,18,1,4,4932
60000,128000,6000,1,18,1,4,4932
61000,227000,6000,1,18,1,4,4932
62000,136000,7000,1,18,1,4,4932
63000,129000,6000,1,18,1,4,4932
64000,121000,5000,1,18,1,4,4932
65000,131000,6000,1,18,1,4,4932
66000,135000,6000,1,18,1,4,4932
67000,131000,7000,1,18,1,4,4932
68000,128000,7000,1,18,1,4,4932
69000,132000,6000,1,18,1,4,4932
70000,141000,7000,1,18,1,4,4965
71000,133000,7000,1,18,1,4,4965
72000,137000,7000,1,18,1,4,4998
73000,139000,6000,1,18,1,4,4998
74000,136000,6000,1,18,1,4,4998
75000,132000,5000,1,18,1,4,4998
76000,126000,6000,1,18,1,4,4998
77000,167000,6000,1,18,1,4,5031
78000,137000,5000,1,18,1,4,5031
79000,136000,6000,1,18,1,4,5031
80000,137000,6000,1,18,1,4,5031
81000,128000,5000,1,18,1,4,5031
82000,205000,6000,1,18,1,4,5031
83000,118000,7000,1,18,1,4,5031
84000,134000,6000,1,18,1,4,5031
85000,122000,5000,1,18,1,4,5102
86000,120000,6000,1,18,1,4,5102
87000,136000,7000,1,18,1,4,5102
88000,134000,6000,1,18,1,4,5102
89000,138000,5000,1,18,1,4,5102
90000,127000,6000,1,18,1,4,5102
91000,131000,7000,1,18,1,4,5102
92000,133000,5000,1,18,1,4,5102
93000,127000,6000,1,18,1,4,5102
94000,132000,6000,1,18,1,4,5102
95000,134000,6000,1,18,1,4,5102
96000,119000,6000,1,18,1,4,5135
97000,135000,6000,1,18,1,4,5135
98000,130000,6000,1,18,1,4,5135
99000,136000,9000,1,18,1,4,5135
100000,123000,5000,1,18,1,4,5135
101000,134000,7000,1,18,1,4,5135
102000,144000,7000,1,18,1,4,5135
103000,132000,6000,1,18,1,4,5135
104000,132000,6000,1,18,1,4,5135
105000,130000,6000,1,18,1,4,5135
106000,129000,5000,1,18,1,4,5135
107000,121000,6000,1,18,1,4,5135
108000,120000,5000,1,18,1,4,5135
109000,138000,6000,1,18,1,4,5135
110000,137000,6000,1,18,1,4,5135
111000,131000,6000,1,18,1,4,5135
112000,137000,6000,1,18,1,4,5135
113000,176000,7000,1,18,1,4,5135
114000,133000,5000,1,18,1,4,5135
115000,147000,7000,1,18,1,4,5135
116000,139000,6000,1,18,1,4,5168
117000,138000,7000,1,18,1,4,5168
118000,137000,6000,1,18,1,4,5168
119000,138000,6000,1,18,1,4,5168
120000,152000,6000,1,18,1,4,5168
121000,120000,5000,1,18,1,4,5168
122000,141000,6000,1,18,1,4,5168
123000,135000,6000,1,18,1,4,5201
124000,149000,6000,1,18,1,4,5201
125000,130000,5000,1,18,1,4,5234
126000,147000,7000,1,18,1,4,5234
127000,139000,6000,1,18,1,4,5234
128000,144000,7000,1,18,1,4,5234
129000,135000,7000,1,18,1,4,5234
130000,135000,7000,1,18,1,4,5234
131000,137000,8000,1,18,1,4,5234
132000,185000,7000,1,18,1,4,5234
133000,135000,7000,1,18,1,4,5234
134000,141000,6000,1,18,1,4,5234
135000,133000,6000,1,18,1,4,5234
136000,141000,7000,1,18,1,4,5234
137000,134000,7000,1,18,1,4,5234
138000,136000,6000,1,18,1,4,5234
139000,149000,7000,1,18,1,4,5234
140000,123000,6000,1,18,1,4,5234
141000,142000,8000,1,18,1,4,5234
142000,136000,7000,1,18,1,4,5234
143000,137000,7000,1,18,1,4,5234
144000,136000,7000,1,18,1,4,5234
145000,141000,7000,1,18,1,4,5234
146000,129000,6000,1,18,1,4,5234
147000,142000,7000,1,18,1,4,5234
148000,124000,6000,1,18,1,4,5234
149000,122000,6000,1,18,1,4,5234
150000,143000,7000,1,18,1,4,5234
151000,180000,15000,1,18,1,4,5234
152000,198000,18000,1,18,1,4,5234
153000,207000,12000,1,18,1,4,5234
154000,211000,12000,1,18,1,4,5234
155000,223000,14000,1,18,1,4,5173
156000,263000,15000,1,18,1,4,5173
157000,257000,12000,1,18,1,4,5173
158000,209000,9000,1,18,1,4,5173
159000,766000,11000,1,18,1,4,5173
160000,228000,13000,1,18,1,4,5173
161000,239000,12000,1,18,1,4,5173
162000,211000,20000,1,18,1,4,5173
163000,218000,12000,1,18,1,4,5173
164000,243000,8582000,1,18,1,4,5173
165000,220000,13000,1,18,1,4,5173
166000,409000,19000,1,18,1,4,5173
167000,245000,19000,1,18,1,4,5173
168000,236000,17000,1,18,1,4,5173
169000,334000,18000,1,18,1,4,5173
170000,223000,18000,1,18,1,4,5173
171000,300000,18000,1,18,1,4,5206
172000,239000,16000,1,18,1,4,5206
173000,245000,14000,1,18,1,4,5206
174000,243000,16000,1,18,1,4,5206
175000,285000,8000,1,18,1,4,5206
176000,148000,7000,1,18,1,4,5206
177000,134000,6000,1,18,1,4,5206
178000,126000,6000,1,18,1,4,5206
179000,124000,6000,1,18,1,4,5206
180000,134000,7000,1,18,1,4,5206
181000,191000,9000,1,18,1,4,5239
182000,122000,6000,1,18,1,4,5239
183000,136000,7000,1,18,1,4,5239
184000,134000,7000,1,18,1,4,5239
185000,188000,7000,1,18,1,4,5239
186000,118000,8000,1,18,1,4,5239
187000,127000,5000,1,18,1,4,5239
188000,150000,7000,1,18,1,4,5272
189000,137000,8000,1,18,1,4,5272
190000,144000,7000,1,18,1,4,5272
191000,138000,7000,1,18,1,4,5305
192000,111000,5000,1,18,1,4,5305
193000,277000,5000,1,18,1,4,5305
194000,99000,6000,1,18,1,4,5305
195000,198000,6000,1,18,1,4,5305
196000,129000,5000,1,18,1,4,5305
197000,141000,7000,1,18,1,4,5305
198000,155000,7000,1,18,1,4,5305
199000,124000,6000,1,18,1,4,5305
200000,140000,7000,1,18,1,4,5305
201000,134000,7000,1,18,1,4,5305
202000,118000,6000,1,18,1,4,5305
203000,121000,5000,1,18,1,4,5305
204000,134000,7000,1,18,1,4,5305
205000,121000,5000,1,18,1,4,5305
206000,117000,6000,1,18,1,4,5305
207000,119000,5000,1,18,1,4,5305
208000,132000,7000,1,18,1,4,5305
209000,165000,8000,1,18,1,4,5305
210000,116000,5000,1,18,1,4,5305
211000,135000,7000,1,18,1,4,5305
212000,130000,6000,1,18,1,4,5305
213000,137000,7000,1,18,1,4,5305
214000,136000,6000,1,18,1,4,5305
215000,136000,7000,1,18,1,4,5305
216000,135000,7000,1,18,1,4,5305
217000,151000,7000,1,18,1,4,5305
218000,122000,6000,1,18,1,4,5305
219000,120000,5000,1,18,1,4,5338
220000,137000,7000,1,18,1,4,5338
221000,122000,6000,1,18,1,4,5338
222000,127000,6000,1,18,1,4,5338
223000,126000,6000,1,18,1,4,5338
224000,135000,5000,1,18,1,4,5338
225000,128000,6000,1,18,1,4,5338
226000,141000,7000,1,18,1,4,5338
227000,117000,6000,1,18,1,4,5338
228000,132000,7000,1,18,1,4,5338
229000,122000,5000,1,18,1,4,5338
230000,129000,7000,1,18,1,4,5338
231000,154000,7000,1,18,1,4,5338
232000,141000,7000,1,18,1,4,5338
233000,136000,7000,1,18,1,4,5338
234000,118000,6000,1,18,1,4,5338
235000,142000,17000,1,18,1,4,5338
236000,231000,7000,1,18,1,4,5338
237000,137000,8000,1,18,1,4,5338
238000,125000,5000,1,18,1,4,5338
239000,125000,7000,1,18,1,4,5338
240000,169000,8000,1,18,1,4,5338
241000,124000,6000,1,18,1,4,5338
242000,148000,7000,1,18,1,4,5338
243000,120000,6000,1,18,1,4,5338
244000,138000,7000,1,18,1,4,5338
245000,130000,7000,1,18,1,4,5338
246000,123000,6000,1,18,1,4,5338
247000,143000,6000,1,18,1,4,5338
248000,152000,7000,1,18,1,4,5338
249000,121000,6000,1,18,1,4,5338
250000,137000,6000,1,18,1,4,5338
251000,131000,6000,1,18,1,4,5338
252000,136000,8000,1,18,1,4,5338
253000,126000,6000,1,18,1,4,5338
254000,134000,7000,1,18,1,4,5338
255000,126000,6000,1,18,1,4,5338
256000,127000,7000,1,18,1,4,5338
257000,132000,6000,1,18,1,4,5338
258000,139000,7000,1,18,1,4,5338
259000,127000,6000,1,18,1,4,5338
260000,141000,9000,1,18,1,4,5338
261000,117000,5000,1,18,1,4,5338
262000,125000,6000,1,18,1,4,5338
263000,140000,8000,1,18,1,4,5338
264000,135000,6000,1,18,1,4,5338
265000,122000,6000,1,18,1,4,5338
266000,129000,6000,1,18,1,4,5338
267000,126000,7000,1,18,1,4,5338
268000,141000,7000,1,18,1,4,5338
269000,130000,7000,1,18,1,4,5338
270000,140000,9000,1,18,1,4,5338
271000,141000,7000,1,18,1,4,5338
272000,144000,7000,1,18,1,4,5338
273000,132000,7000,1,18,1,4,5338
274000,122000,6000,1,18,1,4,5338
275000,122000,6000,1,18,1,4,5338
276000,147000,6000,1,18,1,4,5338
277000,119000,6000,1,18,1,4,5338
278000,123000,6000,1,18,1,4,5371
279000,137000,9000,1,18,1,4,5371
280000,128000,6000,1,18,1,4,5371
281000,130000,5000,1,18,1,4,5371
282000,126000,6000,1,18,1,4,5371
283000,136000,6000,1,18,1,4,5371
284000,133000,6000,1,18,1,4,5371
285000,123000,6000,1,18,1,4,5371
286000,117000,6000,1,18,1,4,5371
287000,117000,6000,1,18,1,4,5371
288000,143000,6000,1,18,1,4,5371
289000,140000,7000,1,18,1,4,5371
290000,121000,6000,1,18,1,4,5371
291000,118000,5000,1,18,1,4,5371
292000,140000,6000,1,18,1,4,5371
293000,135000,7000,1,18,1,4,5371
294000,126000,5000,1,18,1,4,5371
295000,145000,6000,1,18,1,4,5371
296000,126000,5000,1,18,1,4,5371
297000,135000,17000,1,18,1,4,5371
298000,216000,18000,1,18,1,4,5371
299000,229000,11000,1,18,1,4,5371
300000,142000,7000,1,18,1,4,5371

Results are as we expect. There is a logrithmic growth to the PoV size for a balance transfer relative to the number of users in the system.

[shawntabrizi]

/benchmark runtime pallet pallet_balances

[parity-benchapp]

Finished benchmark for branch: shawntabrizi-transfer-increasing-users

Benchmark: Benchmark Runtime Pallet

cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_balances --extrinsic="*" --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/balances/src/weights.rs --template=./.maintain/frame-weight-template.hbs

Results

Pallet: "pallet_balances", Extrinsic: "transfer", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
Median Slopes Analysis

-- Extrinsic Time --

Model:
Time ~= 81.9
µs

Reads = 1
Writes = 1
Min Squares Analysis

-- Extrinsic Time --

Model:
Time ~= 81.9
µs

Reads = 1
Writes = 1
Pallet: "pallet_balances", Extrinsic: "transfer_keep_alive", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
Median Slopes Analysis

-- Extrinsic Time --

Model:
Time ~= 61.07
µs

Reads = 1
Writes = 1
Min Squares Analysis

-- Extrinsic Time --

Model:
Time ~= 61.07
µs

Reads = 1
Writes = 1
Pallet: "pallet_balances", Extrinsic: "set_balance_creating", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
Median Slopes Analysis

-- Extrinsic Time --

Model:
Time ~= 32.25
µs

Reads = 1
Writes = 1
Min Squares Analysis

-- Extrinsic Time --

Model:
Time ~= 32.25
µs

Reads = 1
Writes = 1
Pallet: "pallet_balances", Extrinsic: "set_balance_killing", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
Median Slopes Analysis

-- Extrinsic Time --

Model:
Time ~= 38.51
µs

Reads = 1
Writes = 1
Min Squares Analysis

-- Extrinsic Time --

Model:
Time ~= 38.51
µs

Reads = 1
Writes = 1
Pallet: "pallet_balances", Extrinsic: "force_transfer", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
Median Slopes Analysis

-- Extrinsic Time --

Model:
Time ~= 80.44
µs

Reads = 2
Writes = 2
Min Squares Analysis

-- Extrinsic Time --

Model:
Time ~= 80.44
µs

Reads = 2
Writes = 2

[cheme]

LGTM, small question though, does this bench a full block evaluation or just the balance transfer extrinsic call? (thinking it could be interesting to have multiple transfer in a proof: multiple transfer will share some nodes so it can grow very slightly differently).

[shawntabrizi]

LGTM, small question though, does this bench a full block evaluation or just the balance transfer extrinsic call? (thinking it could be interesting to have multiple transfer in a proof: multiple transfer will share some nodes so it can grow very slightly differently).

This is only one extrinsic. Unfortunately I don't think we can make any assumptions about multiple transactions and how the PoV size will grow in that case. Worst case scenario is that there will only be one transfer extrinsic, and also we cannot have context of other previously executed transaction when doing weights.

[shawntabrizi]

bot merge

[ghost]

Trying merge.