Make SharedTrieCache/LocalTrieCache work with entire state in memory #7682

alexggh · 2025-02-24T09:01:23Z

Part of #6131 (comment), we need to make sure that the TrieCache is able to work with the assumption that more or less the entire current state fits into memory, this can be split into a few parts:

Remove the LocalTrieCache hard coded size and make it configurable from outside.
Decided to have two flavours of building the LocalTrieCache, for the import and block authoring we are using a trusted local cache that grow to hold everything in the block and then propagated everything into the shared trie cache.
[x] Everything from LocalTrieCache needs to be propagated back to the SharedTrieCache in a fast and safe manner, so that it doesn't become a bottleneck. Currently, that is done on drop, with the function holding the ShareTrieCache write lock while promoting all keys. Decided to move this on a separate worker thread, so that the hot-path does not have to wait for drop function to propagate everything from the LocalTrieCache to the SharedTrieCache, the flushing of the worker thread happens after the block production and import happens. Update this is not need because authoring and imports is bounded, so the numbers is not exaggeratedly big, to make the drop too heavy, see numbers here Make SharedTrieCache/LocalTrieCache work with entire state in memory #7682 (comment).

Todo

Stats to prove most of reads/writes hit the shared or the local trie cache.
Unit testing the new changes.

Fixes: #7534

Signed-off-by: Alexandru Gheorghe <[email protected]>

substrate/client/api/src/backend.rs

substrate/client/cli/src/config.rs

substrate/client/db/src/lib.rs

substrate/client/service/src/config.rs

substrate/client/api/src/backend.rs

substrate/client/cli/src/params/import_params.rs

substrate/client/api/src/in_mem.rs

substrate/client/db/src/lib.rs

substrate/primitives/trie/src/cache/shared_cache.rs

AndreiEres

How can we measure the changes?

substrate/primitives/trie/src/cache/shared_cache.rs

substrate/client/service/src/client/client.rs

…_unlimited_v2

Signed-off-by: Alexandru Gheorghe <[email protected]>

…_unlimited_v2

bkchr

Overall looking good, mainly left some comments on smaller changes. One of the main things I don't want to see is that we expose this via CLI nor via a config option. I don't see why we should do this.

substrate/client/cli/src/params/import_params.rs

substrate/client/api/src/backend.rs

substrate/client/db/benches/state_access.rs

substrate/primitives/trie/src/cache/shared_cache.rs

substrate/primitives/trie/src/cache/mod.rs

substrate/primitives/trie/src/cache/shared_cache.rs

substrate/primitives/trie/src/cache/mod.rs

Co-authored-by: Bastian Köcher <[email protected]>

Signed-off-by: Alexandru Gheorghe <[email protected]>

paritytech-workflow-stopper · 2025-04-17T10:43:48Z

All GitHub workflows were cancelled due to failure one of the required jobs.
Failed workflow url: https://github.com/paritytech/polkadot-sdk/actions/runs/14513792430
Failed job name: fmt

Signed-off-by: Alexandru Gheorghe <[email protected]>

…_unlimited_v3

substrate/client/api/src/backend.rs

AndreiEres

Good job!

substrate/primitives/trie/src/cache/mod.rs

substrate/client/api/src/backend.rs

substrate/client/db/src/bench.rs

substrate/utils/frame/benchmarking-cli/src/storage/write.rs

substrate/client/db/src/lib.rs

substrate/primitives/trie/src/cache/shared_cache.rs

Co-authored-by: Bastian Köcher <[email protected]>

Signed-off-by: Alexandru Gheorghe <[email protected]>

alexggh · 2025-04-30T06:56:27Z

Changes are running on kusama-parachain-asset-hub-collator-node-0 and kusama-parachain-asset-hub-collator-node-1 2025-04-28, dashboards for the metrics are here: https://grafana.teleport.parity.io/goto/PLTrT7bNg?orgId=1, I will let it run for a few more days and if no problems arise will merge it.

alexggh · 2025-05-14T13:39:55Z

Changes are running on kusama-parachain-asset-hub-collator-node-0 and kusama-parachain-asset-hub-collator-node-1 2025-04-28, dashboards for the metrics are here: https://grafana.teleport.parity.io/goto/PLTrT7bNg?orgId=1, I will let it run for a few more days and if no problems arise will merge it.

There weren't any problem while running on asset-hub-collators, merging this now.

…7682) Part of #6131 (comment), we need to make sure that the TrieCache is able to work with the assumption that more or less the entire current state fits into memory, this can be split into a few parts: - [x] Remove the LocalTrieCache hard coded size and make it configurable from outside. - [x] Decided to have two flavours of building the LocalTrieCache, for the import and block authoring we are using a trusted local cache that grow to hold everything in the block and then propagated everything into the shared trie cache. - ~[x] Everything from LocalTrieCache needs to be propagated back to the SharedTrieCache in a fast and safe manner, so that it doesn't become a bottleneck. Currently, that is done on drop, with the function holding the ShareTrieCache write lock while promoting all keys. Decided to move this on a separate worker thread, so that the hot-path does not have to wait for drop function to propagate everything from the LocalTrieCache to the SharedTrieCache, the flushing of the worker thread happens after the block production and import happens.~ Update this is not need because authoring and imports is bounded, so the numbers is not exaggeratedly big, to make the drop too heavy, see numbers here #7682 (comment). # Todo - [x] Stats to prove most of reads/writes hit the shared or the local trie cache. - [x] Unit testing the new changes. Fixes: #7534 --------- Signed-off-by: Alexandru Gheorghe <[email protected]> Co-authored-by: Bastian Köcher <[email protected]>

alexggh and others added 10 commits February 24, 2025 10:30

Make local-cache unlimited

0af325f

Signed-off-by: Alexandru Gheorghe <[email protected]>

Remove unrelated changes

f326e8c

Signed-off-by: Alexandru Gheorghe <[email protected]>

Create unlimited cache only in onchain context

7799138

Signed-off-by: Alexandru Gheorghe <[email protected]>

Move writing back to the shared_cache from drop to a worker thread

a8dc8c1

Signed-off-by: Alexandru Gheorghe <[email protected]>

Fix build

c03011a

Signed-off-by: Alexandru Gheorghe <[email protected]>

Add manual trie flush

6be7f6d

Signed-off-by: Alexandru Gheorghe <[email protected]>

Fix on import

94c7f8d

Signed-off-by: Alexandru Gheorghe <[email protected]>

Merge branch 'master' into alexggh/local_cache_unlimited

37a5a87

Some fixes

c90d5eb

Signed-off-by: Alexandru Gheorghe <[email protected]>

Cosmetic changes

96fab66

Signed-off-by: Alexandru Gheorghe <[email protected]>

alexggh marked this pull request as ready for review March 13, 2025 10:37

alexggh changed the title ~~[WIP] Make SharedTrieCache/LocalTrieCache work with entire state in memory~~ Make SharedTrieCache/LocalTrieCache work with entire state in memory Mar 13, 2025

alexggh requested review from AndreiEres, athei, bkchr and sandreim March 13, 2025 10:37

sandreim reviewed Mar 14, 2025

View reviewed changes

bkchr reviewed Mar 20, 2025

View reviewed changes

substrate/client/api/src/backend.rs Outdated Show resolved Hide resolved

substrate/client/cli/src/params/import_params.rs Outdated Show resolved Hide resolved

substrate/client/api/src/in_mem.rs Outdated Show resolved Hide resolved

AndreiEres reviewed Mar 21, 2025

View reviewed changes

substrate/client/db/src/lib.rs Outdated Show resolved Hide resolved

AndreiEres reviewed Mar 21, 2025

View reviewed changes

substrate/primitives/trie/src/cache/shared_cache.rs Outdated Show resolved Hide resolved

AndreiEres reviewed Mar 21, 2025

View reviewed changes

substrate/primitives/trie/src/cache/shared_cache.rs Outdated Show resolved Hide resolved

AndreiEres reviewed Mar 21, 2025

View reviewed changes

substrate/primitives/trie/src/cache/shared_cache.rs Outdated Show resolved Hide resolved

substrate/primitives/trie/src/cache/shared_cache.rs Outdated Show resolved Hide resolved

substrate/client/service/src/client/client.rs Outdated Show resolved Hide resolved

alexggh added 6 commits April 3, 2025 15:40

Merge remote-tracking branch 'origin/master' into alexggh/local_cache…

296e6f5

…_unlimited_v2

Remove unneded changes

ad1aa3f

Signed-off-by: Alexandru Gheorghe <[email protected]>

Add metrics to the trie cache

c267986

Signed-off-by: Alexandru Gheorghe <[email protected]>

Voluntarily give the lock in untrusted paths

596eaa2

Signed-off-by: Alexandru Gheorghe <[email protected]>

Add unittest for cache

b1ad910

Signed-off-by: Alexandru Gheorghe <[email protected]>

Merge remote-tracking branch 'origin/master' into alexggh/local_cache…

39c0ad4

…_unlimited_v2

bkchr reviewed Apr 15, 2025

View reviewed changes

Update substrate/client/api/src/backend.rs

99c758a

Co-authored-by: Bastian Köcher <[email protected]>

alexggh and others added 6 commits April 17, 2025 12:57

Rename default in untrusted

5228fbf

Signed-off-by: Alexandru Gheorghe <[email protected]>

Review suggestions

9e2e204

Signed-off-by: Alexandru Gheorghe <[email protected]>

Move snapshots structs in metrics

f14dbbd

Signed-off-by: Alexandru Gheorghe <[email protected]>

Use a trusted local cache

8245a80

Signed-off-by: Alexandru Gheorghe <[email protected]>

Drop cli flag use_trusted_local_cache

d4504d0

Signed-off-by: Alexandru Gheorghe <[email protected]>

Update basic_authorship.rs

c983020

alexggh added 3 commits April 17, 2025 14:34

Make cargo fmt happy

0906d98

Signed-off-by: Alexandru Gheorghe <[email protected]>

Make taplo happy

418ab9c

Signed-off-by: Alexandru Gheorghe <[email protected]>

Add prdoc

d2661f9

Signed-off-by: Alexandru Gheorghe <[email protected]>

alexggh added the T0-node This PR/Issue is related to the topic “node”. label Apr 17, 2025

Merge remote-tracking branch 'origin/master' into alexggh/local_cache…

858b9dc

…_unlimited_v3

AndreiEres reviewed Apr 25, 2025

View reviewed changes

substrate/client/api/src/backend.rs Show resolved Hide resolved

AndreiEres approved these changes Apr 25, 2025

View reviewed changes

substrate/primitives/trie/src/cache/mod.rs Show resolved Hide resolved

bkchr approved these changes Apr 25, 2025

View reviewed changes

alexggh and others added 3 commits April 25, 2025 20:00

Merge branch 'master' into alexggh/local_cache_unlimited

448b498

Update substrate/primitives/trie/src/cache/shared_cache.rs

9ca77ea

Co-authored-by: Bastian Köcher <[email protected]>

Address review feedback

08c74e5

Signed-off-by: Alexandru Gheorghe <[email protected]>

Merge branch 'master' into alexggh/local_cache_unlimited

f37537a

alexggh added this pull request to the merge queue May 14, 2025

github-merge-queue bot removed this pull request from the merge queue due to failed status checks May 14, 2025

alexggh added this pull request to the merge queue May 14, 2025

Merged via the queue into master with commit 9991b1a May 14, 2025
248 of 249 checks passed

alexggh deleted the alexggh/local_cache_unlimited branch May 14, 2025 15:37

github-actions bot mentioned this pull request Oct 24, 2025

Update polkadot-sdk from stable2503 to stable2506 moondance-labs/tanssi#1340

Closed

manuelmauro mentioned this pull request Nov 26, 2025

Upgrade to Polkadot stable2506 moonbeam-foundation/moonbeam#3488

Merged

Make SharedTrieCache/LocalTrieCache work with entire state in memory #7682

Make SharedTrieCache/LocalTrieCache work with entire state in memory #7682

Uh oh!

Conversation

alexggh commented Feb 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Todo

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AndreiEres left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bkchr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

paritytech-workflow-stopper bot commented Apr 17, 2025

Uh oh!

Uh oh!

AndreiEres left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexggh commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexggh commented May 14, 2025

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

alexggh commented Feb 24, 2025 •

edited

Loading

alexggh commented Apr 30, 2025 •

edited

Loading