This repository was archived by the owner on Nov 15, 2023. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 2.7k
bulk child trie deletion #5280
Closed
Closed
bulk child trie deletion #5280
Changes from 1 commit
Commits
Show all changes
171 commits
Select commit
Hold shift + click to select a range
e97accb
change from cache root pr
cheme d7d69a4
Merge branch 'master' into split_child_payload
cheme cf6393a
Targetted way of putting keyspace.
cheme 2845d0e
changes to state-db
cheme a0532d1
change transaction to be by child trie.
cheme eb5961f
slice index fix, many failing tests.
cheme 67687f8
fix state-db tests
cheme 48df830
vec with multiple entry of a same rc prefixeddb did not make sense,
cheme 9e9a684
Merge branch 'master' into split_child_payload
cheme cb4c4a9
change set to btreemap, seems useless (at least do no solve changetrie
cheme a398b82
moving get_1 to get, state-machine needs rework
cheme 7b26a93
Resolve a bit of child trie
cheme 4d1378f
Merge branch 'split_child_payload' into split_child_payload_map2
cheme f39ce3f
small refact
cheme bc4fe52
Merge branch 'master' into split_child_payload
cheme f27a2ec
Merge branch 'split_child_payload' into split_child_payload_map2
cheme 9f0c600
indent
cheme ecb43d0
Use const of null hash check on BackendStorageRef.
cheme d76a316
Merge branch 'split_child_payload_map2' into split_child_payload
cheme ae29df5
Associated null node hash set to a non optional const.
cheme 6a06c0a
Make ChildInfo borrow of OwnedChildInfo.
cheme 25aaa3a
Removing unsafe cast, using ref_cast asumption for borrow case.
cheme 0d45d85
Borrow approach on OwnedChildInfo and ChildInfo did not make sense
cheme 274a923
Factoring map of children code, before switching key.
cheme 2e47a1d
Switching children key from optional to simple ChildInfo.
cheme 328ed66
Merge branch 'master' into split_child_payload
cheme b07d7ca
fix merge test
cheme e5d7b04
clean todos
cheme 3a71669
fix
cheme c846471
End up removing all keypacedDB from code.
cheme 0b557b6
Clean unused struct and useless child info.
cheme 8483da4
remove todo
cheme 64ffcea
actual touch to keyspace prefixing (fail on wrong code asserted).
cheme 2bc3cb6
self review changes.
cheme 3136355
bump impl version.
cheme f655da1
Merge branch 'master' into split_child_payload
cheme bae6523
calculate size for single operation on usize.
cheme 88ed503
Put keyspace logic in its own struct.
cheme 4eb4676
Restrict top trie in ext to the storage key build from an empty key.
cheme 5151471
Implementation basis for this PR, note that child storage key default
cheme 8715446
Resolve prefix from child_info
cheme b72cd9c
Removed all child prefix, next is putting it back on full storage and on
cheme b636687
Prefixing child storage key where needed.
cheme e88631c
Merge branch 'master' into child_trie_w3_change
cheme e5b525f
Merge branch 'master' into split_child_payload
cheme 728aedf
Remove deprecated check for genesis and fix some test storage builds.
cheme 8329448
Merge branch 'master' into child_trie_w3_change
cheme 864e9ca
Fix contract to stop using the child storage prefix.
cheme c87d19b
Revert changes to state-db and plug keyspace above it for default child
cheme 1f62342
Merge branch 'master' into split_child_payload
cheme 8b901a2
Update client-db benches.
cheme f6efdad
Bump runtime impl version.
cheme c683198
Merge branch 'master' into split_child_payload
cheme 93de960
fix new code
cheme 2224e2f
Merge branch 'master' into child_trie_w3_change
cheme c04ba95
Fixing merge.
cheme ffaf9f5
Switch back to using prefix on host function, remove child_id and
cheme 87bd97c
fix polka ref issue.
cheme 17cf613
Switching back to unprefixed in child info (all temporary struct
cheme b3ccc93
bump version
cheme afe85ae
fix tabs and doc.
cheme befe325
Merge branch 'master' into child_trie_w3_change
cheme ed480fa
Apply more consistant naming 'storage_key' instead of
cheme e103c2a
Update primitives/storage/src/lib.rs
cheme 958b626
use prefixed storage key in change trie
cheme 26f8000
Merge branch 'child_trie_w3_change' of github.com:cheme/polkadot into…
cheme f471d56
renaming 'default_child_trie_root' to 'empty_child_trie_root'
cheme 02ff227
apply some renaming, rpc change are really ugly, will revert them
cheme 8fcc112
Switch back to using child_type for rpc (and light client).
cheme b7b347a
Merge branch 'master' into child_trie_w3_change
cheme 3869774
Merge branch 'master' into child_trie_w3_change
cheme 4ebc204
Merge branch 'master' into child_trie_w3_change
cheme 74aa3f8
bump runtime version
cheme dc023d1
merge #4857
cheme 4c40ea7
Resolve merging change of api from #4857
cheme b6b70f3
revert rpc related default renaming.
cheme d563278
fix slice indexing for child root
cheme a85bb38
Switch back to prefixed storage key for rpc.
cheme 5f98fbe
rpc error discrepancy
cheme 320e375
Merge branch 'master' into child_trie_w3_change
cheme f05a576
Merge branch 'child_trie_w3_change' into split_child_payload
cheme d85dec2
Fix test
cheme 9f704c9
use correct parameter in overlay and fix change trie.
cheme c82edbd
Merge branch 'master' into child_trie_w3_change
cheme 7c5d546
bump version
cheme 19ab06b
Merge branch 'master' into child_trie_w3_change
cheme 152f04f
Merge branch 'master' into child_trie_w3_change
cheme e959e0d
Merge branch 'child_trie_w3_change' into split_child_payload
cheme eff1d8c
keeping inner hasher, it is needed at state-machine level.
cheme 80136e3
Define child change enum.
cheme 8249a9f
Key delta in child change too.
cheme 6f0a01b
remove storage collection from the change, empty collection is not an
cheme 1c97b84
ChildChangeSet overlay struct
cheme daf0a7a
fix client
cheme 00ece8f
fix client
cheme 3a2118e
storage_cache changes
cheme 676f2d0
rocksdb patch for prefix delete updated
cheme 07ec221
unprefixed storage key is smaller than random one.
cheme 67e850e
notification and tests
cheme 8160e20
change try_update to skip update on deleted child
cheme 97bbb55
assimilate delete for child change
cheme fc46e62
overlay filters
cheme 61b1926
in progress, putting child change in transaction (types wrong), next
cheme 921e9d1
Keep change trie delta for drop child.
cheme cd9d47c
finish adding child change
cheme 2e7bc65
Put bulk delete into top change child trie info, but currently not
cheme b0e0af8
remove done todos
cheme c70ef19
Compatibility with old version journal.
cheme b1994e7
prune test, and fixes.
cheme 0036fd3
Merge branch 'master' into child_trie_w3_change
cheme 29254aa
missing merge
cheme 72cfb6c
Merge branch 'child_trie_w3_change' into split_child_payload
cheme 594fdc8
Merge branch 'split_child_payload' into child_delete_2020
cheme baacffa
fix bench
cheme 1471c0f
Merge branch 'split_child_payload' into child_delete_2020
cheme 8e1fabc
bump version
cheme 160d2ea
Restore previous cli api (will need deprecation in a v2 of rpc), split
cheme c76b433
Merge branch 'child_trie_w3_change' into split_child_payload
cheme b07fd39
Merge branch 'split_child_payload' into child_delete_2020
cheme 63cf6bd
remove old rpc
cheme 5ed450b
Merge branch 'master' into child_trie_w3_change
cheme f21606f
review change.
cheme 1b2a30c
Using `PrefixedStorageKey` type for child storage key that are prefixed.
cheme 750cdd8
Fix rpc test.
cheme 32e27c9
Merge branch 'master' into child_trie_w3_change
cheme cc65f44
bump spec version
cheme c20cef9
Merge branch 'master' into child_trie_w3_change
cheme 797ad34
Merge branch 'master' into child_trie_w3_change
cheme 7e4b405
update head for the rocksdb delete.
cheme f1b23df
Apply review suggestion
cheme 9651dd1
Merge branch 'master' into child_trie_w3_change
cheme c631a46
Fix unrelated warning for CI
cheme 217c19b
Merge branch 'master' into child_trie_w3_change
cheme cf7d90e
Merge branch 'child_trie_w3_change' into split_child_payload
cheme 7c8f393
companion fix
cheme b2240f1
Update .maintain/gitlab/check_polkadot_companion_build.sh
cheme f3b9f23
Fix compilation errors.
cheme 63943ee
last attempt
cheme 0d15f6f
Merge branch 'split_child_payload' into child_delete_2020
cheme 784cb22
Forcing companion pr.
cheme 4897109
fixing merge.
cheme 4bbb24d
revert ci changes.
cheme 50c3dd7
Merge branch 'master' into child_trie_w3_change
cheme a7c16eb
Merge branch 'child_trie_w3_change' of github.com:cheme/substrate int…
cheme ca40040
Merge branch 'master' into child_trie_w3_change
cheme 226f7cd
name of children in chain spec change.
cheme 5ab115e
Merge branch 'master' into child_trie_w3_change
cheme 1ad4c5a
Merge branch 'master' into child_trie_w3_change
cheme 94b78f3
Merge branch 'child_trie_w3_change' into split_child_payload
cheme 5938d86
remove terminal space
cheme 30e8192
Merge branch 'master' into child_trie_w3_change
cheme bc2a198
sp-io documentation changes.
cheme 83ff4a0
Merge branch 'master' into child_trie_w3_change
cheme ecb1c39
Merge branch 'master' into child_trie_w3_change
cheme 619b454
Retain compatibility with network protocol.
cheme fa52a8c
Revert "Retain compatibility with network protocol."
cheme 167f20c
Merge branch 'master' into child_trie_w3_change
cheme 928fac0
Merge branch 'master' into child_trie_w3_change
gavofyork 69ed118
fix renamed field related error
cheme b79bd84
Merge branch 'master' into child_trie_w3_change
cheme ae3a3e1
Merge branch 'child_trie_w3_change' into split_child_payload
cheme ee1f768
Merge branch 'master' into split_child_payload
cheme 1b3e752
Merge branch 'split_child_payload' into child_delete_2020
cheme 52aa0f2
prototyping changes
cheme eb8a25d
would need rollback, this is going in a wrong direction
cheme c48ea04
might be ok design (could use &mut dyn ptr too, but this seems less
cheme 8e63633
Ok design (no trait an pure fn ptr).
cheme 7afbe6a
add root to child change
cheme 03dae51
got the method test in parity-db non rc version (the reference db trick
cheme 329983d
fix next storage key logic
cheme 12f4229
Merge branch 'child_delete_2020' of github.com:cheme/polkadot into ch…
cheme File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
remove old rpc
- Loading branch information
commit 63cf6bd01d20e06009c6c96814d9ef1984418cb1
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a breaking change to the RPC, we should probably plan some transition or add versioning, os that the JS libs can figure out which API version it's operating on.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is more related to this PR https://github.com/paritytech/substrate/pull/4857/files#r394314720 , there was a PR pending from @Stefie on the js side, I think the initial idea was that the api was never really well defined and do it in a breaking way, but the more time passes the more probable it will be needed to be versioned, I will look at how it can be versioned (I feel like as long as this makes it to 2.0 versioning is not needed but I may be wrong and #4857 may not make it to 2.0).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fair enough! We have a RPC call that returns a list of all available RPC methods (and their versions), but it's hardcoded to
1for now. We don't really have a good mechanism to control that, but it's definitely possible.If the API is not available on JS side yet (or not being used extensively) then I suppose it's probably fine to change without version change before 2.0), but if not then we need some migration path. I remember that JS team (notably @jacogr) had a really hard time maintaining different compatibility mechanisms.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, the issue is we cannot "just break" - the pending JS PR will not go in at all as it stands. We support different chains and cannot detect among them what params there are and what params there are not.
So as it stands, we will break some and not others. Sadly, not something that is quite acceptable. Needs a fix.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And by "fix", I mean the simplest is actually just a rename. No other magic required.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Versioning will be better, I will have a look later. It is a tricky question to know how long deprecated interface would need support, maybe there is something to do by putting old api in some compatibility crate to avoid new project using them.
(Unlike runtime api there is no need to keep supporting old api forever)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My PR is now a month old, now, so I also wouldn't just merge it anymore. I haven't touched it since I opened it. I'll close it to avoid confusion.