Fix benchmarks: set cache_size to higher value

[NikVolf]

No description provided.

[NikVolf]

/bench import

[parity-benchapp]

Finished benchmark for branch: nv-fix-bench-setup

Benchmark: Import Benchmark (random transfers)

Master: 697.50 ms
Branch: 20.91 ms
Change: -97.00%

[cecton]

It feels weird that this value (128) is hardcoded at so many different places.

[NikVolf]

hmm.. according to parity-common/kvdb-rocksdb, cache size is in MiB, but it does not explain performance degradation by setting it to 128

probably something is getting rounded to 0 somewhere

[cecton]

I mean, maybe we should put a const somewhere with the value (or parametrize it in sc_service::Configuration)... because right now if we need to change the value we have to do it in 5 different places.

[NikVolf]

/bench import

[parity-benchapp]

Finished benchmark for branch: nv-fix-bench-setup

Benchmark: Import Benchmark (random transfers)

Master: 697.35 ms
Branch: 19.54 ms
Change: -97.20%

[NikVolf]

I mean, maybe we should put a const somewhere with the value (or parametrize it in sc_service::Configuration)... because right now if we need to change the value we have to do it in 5 different places.

this is rather out of scope of this pr, I want to resolve issue with benchmarks reporting degradation of 10x caused by #5271

And at the moment, I am not sure that only benchmarks are affected

[bkchr]

Before the linked pr the value was set to None. This meant we used the default size per column that kvdb sets. And that is way higher as 128MB per column.

If we set 128MB, it means that all columns needs to share this and the state gets 90% of this.

[NikVolf]

Before the linked pr the value was set to None. This meant we used the default size per column that kvdb sets. And that is way higher as 128MB per column.

If we set 128MB, it means that all columns needs to share this and the state gets 90% of this.

I think there is error in calculations (rounding) somewhere

Now setting cache-size = 128 basically disables rocksdb cache and leads to this 10x-20x degradation ^^^

[bkchr]

Yes we round these numbers: https://github.com/paritytech/substrate/blob/master/client/db/src/utils.rs#L236

However, this is not your problem. You have the problem that I explained above. 128MB is for all columns, while the state column gets 90% of it.

Before when the value was None, it set the default value here in kvdb.

[NikVolf]

Well now if we run --db-cache=64 we get 0 Mb cache for all columns except state column, which is not correct (cache should never be disabled for any column, and 0 disables it).

But it still does not explain such degradation with 128 value.

[NikVolf]

You have the problem that I explained above.

This does not explain degradation. Such degradation is only possible when cache is disabled.

[bkchr]

Why only when the cache is disabled? Even when you have much less cache, the performance could degrade heavily as stuff is constantly paged out of cache. A small cache leads to the same behaviors as a disabled cache.

[NikVolf]

Why only when the cache is disabled? Even when you have much less cache, the performance could degrade heavily as stuff is constantly paged out of cache. A small cache leads to the same behaviors as a disabled cache.

True, but have no idea how to profile it, I think node performance might be affected as well with these settings. One or several of the column cache might also be too small. Probably not the state one, since it takes 90%. I think setting it to some minimum value (5MiB) might help.

[bkchr]

We set the value 128MB after some testing. The point was to have validators running without using abusive amounts of memory.
When syncing you need to increase this value to get better performance, but this is expected.

[NikVolf]

We set the value 128MB after some testing. The point was to have validators running without using abusive amounts of memory.
When syncing you need to increase this value to get better performance, but this is expected.

I know, still there is no explanation for some observations.

[NikVolf]

/bench import

[parity-benchapp]

Finished benchmark for branch: nv-fix-bench-setup

Benchmark: Import Benchmark (random transfers)

Master: 693.28 ms
Branch: 697.59 ms
Change: +0.62%

[NikVolf]

/bench import

[parity-benchapp]

Finished benchmark for branch: nv-fix-bench-setup

Benchmark: Import Benchmark (random transfers)

Master: 693.53 ms
Branch: 694.02 ms
Change: +0.07%

[NikVolf]

Ok, observations show that this is state cache that is not enough (for benchmarks at least). This is probably indication of some weird behaviour when specifying memory_budget for kvdb-rocksdb. Because before we didn't specify it and should've got 128Mb per for state column, now we specify 256Mb that should get more than 128Mb for state column and still we get regression (which is eliminated by specifying 512 🤷‍♀️).

[NikVolf]

/bench import

[parity-benchapp]

Finished benchmark for branch: nv-fix-bench-setup

Benchmark: Import Benchmark (random transfers)

Master: 693.30 ms
Branch: 19.53 ms
Change: -97.18%