Remove discarded blocks and states from database by default #11983

hzy1919 · 2022-08-05T09:28:57Z

Fixes #11912

1.Add state_pruning's sub-param archive-canonical in sc-cli.
2.Change blocks_pruning to be like state_pruning : use the string archive for BlocksPruning::KeepAll and archive-canonical(and default) for BlocksPruning::KeepFinalized and then a number of BlocksPruning::Some(x).

polkadot companion: paritytech/polkadot#5992
cumulus companion: paritytech/cumulus#1613

Closes: #12094

2.Make PruningMode's default value to ArchiveCanonical.

…tabase

arkpar · 2022-08-09T17:43:48Z

client/cli/src/params/pruning_params.rs

 			.as_ref()
 			.map(|s| match s.as_str() {
 				"archive" => Ok(PruningMode::ArchiveAll),
+				"canonical" => Ok(PruningMode::ArchiveCanonical),


"canonical" is more of an internal concept, and not a user friendly term. Let's call it "final" or "finalized"

Ok , Let's set it "final".

arkpar · 2022-08-09T17:45:13Z

There's no need to change the defult pruning mode. It should remain at 256 block history.

arkpar · 2022-08-09T17:49:08Z

Regarding #11912, adding a CLI for ArchiveCanonical only fixes it half way. The major piece of work required here is to remove block bodies, and not just state. This part is a bit more involved.

hzy1919 · 2022-08-10T00:36:35Z

Regarding #11912, adding a CLI for ArchiveCanonical only fixes it half way. The major piece of work required here is to remove block bodies, and not just state. This part is a bit more involved.

Well , I'll rethink where to start...

…tabase

stale · 2022-09-09T01:05:57Z

Hey, is anyone still working on this? Due to the inactivity this issue has been automatically marked as stale. It will be closed if no further activity occurs. Thank you for your contributions.

… Corresponds to `blocks_pruning 0` in CLI . 2.Change value `All` to `AllFinalized` in `enum BlocksPruning` and make it to keep full finalized block history.

…tabase

client/db/src/lib.rs

arkpar · 2022-09-09T10:53:52Z

Looks good with some minor comments. One addition that's also required here, is that prune_block function also deletes justification data.

		utils::remove_from_db(
			transaction,
			&*self.storage.db,
			columns::KEY_LOOKUP,
			columns::JUSTIFICATIONS,
			id,
		)?;

hzy1919 · 2022-09-10T08:18:20Z

Looks good with some minor comments. One addition that's also required here, is that prune_block function also deletes justification data.

Done.

…rsation.

cheme

Looks good to me. Just got some question suggestion for the Cli and maybe using different names, but did not really have any good ideas.

cheme · 2022-09-13T08:07:33Z

client/cli/src/params/pruning_params.rs

 	/// otherwise, the state can be kept for all of the blocks (i.e 'archive'),
 	/// or for all of the canonical blocks (i.e 'archive-canonical').
 	#[clap(alias = "pruning", long, value_name = "PRUNING_MODE")]
 	pub state_pruning: Option<String>,


The doc for this param should be updated with the new final option.

cheme · 2022-09-13T08:08:49Z

client/cli/src/params/pruning_params.rs

 			.as_ref()
 			.map(|s| match s.as_str() {
 				"archive" => Ok(PruningMode::ArchiveAll),
+				"final" => Ok(PruningMode::ArchiveCanonical),


In the doc there is also a "archive-canonical" mention that did not look linked, I would maybe use it instead of "final".
But tbh I did not find a good name. Since it is the new default it should be shorter or more attractive than 'archive', but 'final' is not really saying much to me.
If anybody got other ideas?
(ideally former 'archive' becomes 'archive-all' and 'final' wil be 'archive', but it breaks compatibility)

Yeah, it should be called archive-canonical

cheme · 2022-09-13T08:15:26Z

client/cli/src/params/pruning_params.rs

 	/// Specify the number of finalized blocks to keep in the database.
 	///
-	/// Default is to keep all blocks.
+	/// Default is to keep all of finalized blocks.


Here we should add that 0 keep all blocks.
Generally this is not very intuitive.
Maybe we should do something like state_pruning: use the same string: 'archive' for 'keep_all' and 'final' fo 'keep_finalized'. (subject to change of name too).

Hmm,,It's really not very intuitive that 0 keep all blocks. IMO it's a compromise approach now.
I agree that change blocks_pruning to be like state_pruning.

cheme · 2022-09-13T08:21:12Z

client/db/src/lib.rs

+			columns::KEY_LOOKUP,
+			columns::JUSTIFICATIONS,
+			id,
+		)?;


May be worth mentioning in release note that justification are now pruned with block (additionally to the new parameters).
But looks like a good thing to me 👍

bkchr

Generally looking good. However, there is no test.

client/db/src/lib.rs

bkchr · 2022-09-13T08:54:03Z

client/cli/src/params/pruning_params.rs

 			.as_ref()
 			.map(|s| match s.as_str() {
 				"archive" => Ok(PruningMode::ArchiveAll),
+				"final" => Ok(PruningMode::ArchiveCanonical),


Yeah, it should be called archive-canonical

bkchr · 2022-09-13T09:00:36Z

client/cli/src/params/pruning_params.rs

 	/// Get the block pruning value from the parameters
 	pub fn blocks_pruning(&self) -> error::Result<BlocksPruning> {
 		Ok(match self.blocks_pruning {
+			Some(0) => BlocksPruning::KeepAll,


Okay that is relative random. That this switches to KeepAll when you give it 0.

Maybe we should not expose KeepAll at all. Or we really need to introduce some new cli flag.

what do you think about that change blocks_pruning to be like state_pruning : use the string keep-all for BlocksPruning::KeepAll and keep-finalized(and default) for BlocksPruning::KeepFinalized ?
And IMO ,It may help to keep the style consistent with state_pruning and be better to be understood .

Using same naming would be good IMO.

Using same naming would be good IMO.

That also be OK .

Let's change blocks_pruning to be like state_pruning : use the string archive for BlocksPruning::KeepAll and archive-canonical(and default) for BlocksPruning::KeepFinalized ? @bkchr @arkpar

And then a number of BlocksPruning::Some(x)?

Sounds good.

hzy1919 · 2022-09-14T02:42:35Z

Looks good. Could also use a companion PR to the polkadot repo. See here: https://github.com/paritytech/substrate/blob/master/docs/CONTRIBUTING.adoc#updating-polkadot-as-well

yeah, I added before. :D

Co-authored-by: Bastian Köcher <[email protected]>

…tabase

client/cli/src/params/pruning_params.rs

bkchr · 2022-09-24T19:44:58Z

client/cli/src/params/pruning_params.rs

 	/// NOTE: only finalized blocks are subject for removal!
 	#[clap(alias = "keep-blocks", long, value_name = "COUNT")]
-	pub blocks_pruning: Option<u32>,
+	pub blocks_pruning: Option<String>,


You also need to document the options available.

client/cli/src/params/pruning_params.rs

bkchr · 2022-09-24T21:39:00Z

client/db/src/lib.rs

+	/// Same function as `new_test_with_tx_storage`,just change the type of `blocks_pruning` to
+	/// `BlocksPruning`.
+	#[cfg(any(test, feature = "test-helpers"))]
+	pub fn new_test_with_tx_storage_2(


Can we not just change new_test_with_tx_storage to take BlocksPruning by default?

Hmm,,I think that will work too.

Co-authored-by: Bastian Köcher <[email protected]>

…tabase

bkchr · 2022-09-26T07:42:17Z

Ty @hzy1919

bkchr · 2022-09-26T07:42:23Z

bot merge

paritytech-processbot · 2022-09-26T07:42:32Z

Error: "Check reviews" status is not passing for paritytech/cumulus#1613

bkchr · 2022-09-26T07:46:50Z

bot merge

hzy1919 · 2022-09-27T10:28:31Z

Ty @hzy1919

You're welcome. @bkchr

…ch#11983) * 1.Add pruning param "canonical" in sc-cli. 2.Make PruningMode's default value to ArchiveCanonical. * Update tests in sc-state-db. * Update tests in sc-state-db. * 1.Add a new value `AllWithNonFinalized` in `enum BlocksPruning` which Corresponds to `blocks_pruning 0` in CLI . 2.Change value `All` to `AllFinalized` in `enum BlocksPruning` and make it to keep full finalized block history. * Make some corresponding adjustments based on the content in the conversation. * Update client/db/src/lib.rs Co-authored-by: Bastian Köcher <[email protected]> * Apply suggestions from code review. * 1.Change `blocks_pruning` to be like `state_pruning` . * Fmt and add some doc. * Update client/cli/src/params/pruning_params.rs Co-authored-by: Bastian Köcher <[email protected]> * Update client/cli/src/params/pruning_params.rs Co-authored-by: Bastian Köcher <[email protected]> * Update doc. * Change `new_test_with_tx_storage` to take `BlocksPruning`. * Fmt Co-authored-by: Bastian Köcher <[email protected]>

Polkadot-Forum · 2023-03-09T19:30:31Z

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/polkadot-release-analysis-v0-9-39/2277/1

hzy1919 added 4 commits August 5, 2022 17:25

1.Add pruning param "canonical" in sc-cli.

bc5ba8e

2.Make PruningMode's default value to ArchiveCanonical.

Merge branch 'master' into Remove-discarded-blocks-and-states-from-da…

d762b63

…tabase

Update tests in sc-state-db.

e425634

Merge branch 'master' into Remove-discarded-blocks-and-states-from-da…

70b21b8

…tabase

hzy1919 changed the title ~~Remove discarded blocks and states from database~~ Remove discarded blocks and states from database by default Aug 8, 2022

hzy1919 marked this pull request as ready for review August 8, 2022 09:21

hzy1919 added 2 commits August 8, 2022 18:18

Merge branch 'master' into Remove-discarded-blocks-and-states-from-da…

124f586

…tabase

Update tests in sc-state-db.

e2c5674

hzy1919 marked this pull request as draft August 9, 2022 11:43

arkpar reviewed Aug 9, 2022

View reviewed changes

arkpar requested a review from bkchr August 9, 2022 17:49

hzy1919 added 3 commits August 19, 2022 09:50

Merge branch 'master' into Remove-discarded-blocks-and-states-from-da…

7bff3cb

…tabase

Merge branch 'master' into Remove-discarded-blocks-and-states-from-da…

da0c426

…tabase

Merge branch 'master' into Remove-discarded-blocks-and-states-from-da…

c519937

…tabase

stale bot added the A5-stale Pull request did not receive any updates in a long time. No review needed at this stage. Close it. label Sep 9, 2022

hzy1919 added 2 commits September 9, 2022 14:59

1.Add a new value AllWithNonFinalized in enum BlocksPruning which…

3f03c82

… Corresponds to `blocks_pruning 0` in CLI . 2.Change value `All` to `AllFinalized` in `enum BlocksPruning` and make it to keep full finalized block history.

Merge branch 'master' into Remove-discarded-blocks-and-states-from-da…

940d9db

…tabase

stale bot removed the A5-stale Pull request did not receive any updates in a long time. No review needed at this stage. Close it. label Sep 9, 2022

hzy1919 marked this pull request as ready for review September 9, 2022 07:09

arkpar reviewed Sep 9, 2022

View reviewed changes

client/db/src/lib.rs Show resolved Hide resolved

arkpar reviewed Sep 9, 2022

View reviewed changes

client/db/src/lib.rs Show resolved Hide resolved

arkpar reviewed Sep 9, 2022

View reviewed changes

client/db/src/lib.rs Show resolved Hide resolved

Make some corresponding adjustments based on the content in the conve…

458e078

…rsation.

hzy1919 mentioned this pull request Sep 10, 2022

Companion for substrate#11983. paritytech/polkadot#5992

Merged

cheme approved these changes Sep 13, 2022

View reviewed changes

bkchr reviewed Sep 13, 2022

View reviewed changes

hzy1919 and others added 7 commits September 14, 2022 10:52

Update client/db/src/lib.rs

a228701

Co-authored-by: Bastian Köcher <[email protected]>

Apply suggestions from code review.

6ae25a1

Merge branch 'master' into Remove-discarded-blocks-and-states-from-da…

54a5a2b

…tabase

1.Change blocks_pruning to be like state_pruning .

9e3e8eb

Merge branch 'master' into Remove-discarded-blocks-and-states-from-da…

ac58113

…tabase

Fmt and add some doc.

4c3810c

Merge branch 'master' into Remove-discarded-blocks-and-states-from-da…

f4ea09b

…tabase

bkchr reviewed Sep 24, 2022

View reviewed changes

hzy1919 and others added 6 commits September 25, 2022 21:18

Update client/cli/src/params/pruning_params.rs

2b0c25c

Co-authored-by: Bastian Köcher <[email protected]>

Update client/cli/src/params/pruning_params.rs

6cbc62e

Co-authored-by: Bastian Köcher <[email protected]>

Update doc.

df5ffec

Change new_test_with_tx_storage to take BlocksPruning.

cb62c23

Merge branch 'master' into Remove-discarded-blocks-and-states-from-da…

79afe8b

…tabase

Fmt

3dff0f0

bkchr approved these changes Sep 26, 2022

View reviewed changes

paritytech-processbot bot merged commit a0ec652 into paritytech:master Sep 26, 2022

4meta5 mentioned this pull request Oct 17, 2022

Update substrate/polkadot from v0.9.29 to v0.9.30 moonbeam-foundation/moonbeam#1874

Closed

1 task

hzy1919 mentioned this pull request Oct 26, 2022

Member request polkadot-fellows/seeding#53

Closed

github-actions bot mentioned this pull request Nov 10, 2022

Update substrate/polkadot/cumulus from v0.9.29 to v0.9.31 moonbeam-foundation/moonbeam#1942

Closed

github-actions bot mentioned this pull request Nov 23, 2022

Update substrate/polkadot/cumulus from v0.9.29 to v0.9.32 moonbeam-foundation/moonbeam#1965

Closed

Remove discarded blocks and states from database by default #11983

Remove discarded blocks and states from database by default #11983

Uh oh!

Conversation

hzy1919 commented Aug 5, 2022 • edited by bkchr Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arkpar commented Aug 9, 2022

Uh oh!

arkpar commented Aug 9, 2022

Uh oh!

hzy1919 commented Aug 10, 2022

Uh oh!

stale bot commented Sep 9, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

arkpar commented Sep 9, 2022

Uh oh!

hzy1919 commented Sep 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cheme left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bkchr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hzy1919 commented Sep 14, 2022

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bkchr commented Sep 26, 2022

Uh oh!

bkchr commented Sep 26, 2022

Uh oh!

paritytech-processbot bot commented Sep 26, 2022

Uh oh!

bkchr commented Sep 26, 2022

Uh oh!

hzy1919 commented Sep 27, 2022

Uh oh!

Polkadot-Forum commented Mar 9, 2023

hzy1919 commented Aug 5, 2022 •

edited by bkchr

Loading

hzy1919 commented Sep 10, 2022 •

edited

Loading

cheme left a comment •

edited

Loading