DatabaseSource::Auto #9500

debris · 2021-08-04T17:27:04Z

This pr introduces a DatabaseSource enum variant called Auto. (part of #9201)

polkadot companion: paritytech/polkadot#3590

Auto database backend is used for backward compatibility. It detects if rocksdb database already exists and uses it. Otherwise defaults to paritydb.

changes:

bump kvdb-rocksdb to 0.13
bump parity-db to 0.3
rename DatabaseSettingsSrc to DatabaseSource and remove it's alias service::config::DatabaseConfig
introduce new DatabaseSource variant Auto which always attempts to open existing rocksdb and if the database does not exist (or fails to open), it attempts to open (or create new) paritydb
move fn DatabaseSource::supports_ref_counting to trait Database and add default impl that returns false
added tests

…ait to make it work correctly for all types of dbs

…ly auto start existing database

client/db/src/lib.rs

client/db/src/parity_db.rs

Cargo.lock

client/db/src/utils.rs

arkpar · 2021-08-04T17:51:25Z

Just want to make sure default CLI options is still RocksDb. We don't want to activate parity-db by default for everyone yet, before #9201 is complete. Needs some testing on validators first.

…handle case when database is not enabled at the compile time

…ation

…db are disabled

client/db/src/lib.rs

debris · 2021-08-06T14:52:23Z

Just want to make sure default CLI options is still RocksDb. We don't want to activate parity-db by default for everyone yet, before #9201 is complete. Needs some testing on validators first.

Default cli option remains unchanged rocksdb

client/db/src/upgrade.rs

… suggestion

arkpar · 2021-08-07T15:02:08Z

client/db/src/lib.rs

+			//
+			// IIUC this is needed for polkadot to create its own dbs, so until it can use parity db
+			// I would think rocksdb, but later parity-db.
+			DatabaseSource::Auto { paritydb_path, .. } => Some(&paritydb_path),


I think this does not matter much for now. There's a separate item in #9201 to resolve polkadot db path properly.

debris · 2021-08-09T12:52:52Z

gitlab-unleash-check is failure is unrelated to this pr

arkpar · 2021-08-09T13:22:23Z

bot merge

ghost · 2021-08-09T13:22:27Z

Trying merge.

This was overseen in: #9500

debris added 4 commits August 4, 2021 16:09

implement "auto" database backend in client/db, in progress, #9201

9d76e4e

move fn supports_ref_counting from DatabaseSource enum to Database tr…

3f4305f

…ait to make it work correctly for all types of dbs

update kvdb_rocksdb to 0.13 and use it's new config feature to proper…

efe4fa5

…ly auto start existing database

Merge branch 'master' into auto_db

adbd471

debris added A0-please_review Pull request needs code review. B3-apinoteworthy C1-low PR touches the given topic and has a low impact on builders. D2-breaksapi labels Aug 4, 2021

debris requested review from arkpar and cheme August 4, 2021 17:27

ordian reviewed Aug 4, 2021

View reviewed changes

client/db/src/lib.rs Show resolved Hide resolved

arkpar reviewed Aug 4, 2021

View reviewed changes

client/db/src/parity_db.rs Outdated Show resolved Hide resolved

ordian reviewed Aug 4, 2021

View reviewed changes

Cargo.lock Outdated Show resolved Hide resolved

arkpar reviewed Aug 4, 2021

View reviewed changes

client/db/src/utils.rs Outdated Show resolved Hide resolved

debris added 13 commits August 6, 2021 10:59

tests for auto database reopening

1897664

introduce OpenDbError to cleanup opening database error handling and …

253a487

…handle case when database is not enabled at the compile time

Merge branch 'master' into auto_db

d76ae74

cargo fmt strings again

4279aae

cargo fmt strings again

395f017

rename DataSettingsSrc to fix test compilation

6c4401a

fix the call to the new kvdb-rocksdb interdace in tests to fix compil…

681bbeb

…ation

simplify OpenDbError and make it compile even when paritydb and rocks…

91fd53d

…db are disabled

cargo fmt

b0a4b9d

fix compilation without flag with-parity-db

65cd74a

fix unused var compilation warning

6d6a2f7

support different paths for rocksdb and paritydb in DatabaseSouce::Auto

5729ee5

support "auto" database option in substrate cli

119f50f

debris commented Aug 6, 2021

View reviewed changes

client/db/src/lib.rs Outdated Show resolved Hide resolved

arkpar reviewed Aug 6, 2021

View reviewed changes

client/db/src/upgrade.rs Outdated Show resolved Hide resolved

debris added 2 commits August 6, 2021 19:22

enable Lz4 compression for some of the parity-db colums as per review…

c7fb48e

… suggestion

applied review suggestions

63ca796

debris mentioned this pull request Aug 7, 2021

companion pr for substrate:auto_db paritytech/polkadot#3590

Merged

arkpar reviewed Aug 7, 2021

View reviewed changes

arkpar approved these changes Aug 7, 2021

View reviewed changes

cheme approved these changes Aug 7, 2021

View reviewed changes

arkpar added the D3-trivial 🧸 PR contains trivial changes in a runtime directory that do not require an audit label Aug 9, 2021

ghost merged commit b9d86e1 into master Aug 9, 2021

ghost deleted the auto_db branch August 9, 2021 13:22

JoshOrndorff added a commit to moonbeam-foundation/frontier that referenced this pull request Aug 24, 2021

revert changes related to paritytech/substrate#9500

cc63802

JoshOrndorff added a commit to moonbeam-foundation/frontier that referenced this pull request Sep 16, 2021

revert changes related to paritytech/substrate#9500

7fed1ae

JoshOrndorff added a commit to moonbeam-foundation/frontier that referenced this pull request Sep 28, 2021

revert changes related to paritytech/substrate#9500

a102735

JoshOrndorff added a commit to moonbeam-foundation/frontier that referenced this pull request Sep 28, 2021

revert changes related to paritytech/substrate#9500

47c682a

bkchr added a commit that referenced this pull request Oct 8, 2021

Use correct db path for parity-db

c913527

This was overseen in: #9500

bkchr mentioned this pull request Oct 8, 2021

Use correct db path for parity-db #9971

Merged

bkchr added a commit that referenced this pull request Oct 8, 2021

Use correct db path for parity-db (#9971)

0e22009

This was overseen in: #9500

bkchr added a commit that referenced this pull request Oct 8, 2021

Use correct db path for parity-db (#9971)

9ee94fa

This was overseen in: #9500

github-actions bot mentioned this pull request Oct 11, 2021

Update substrate/polkadot/cumulus from v0.9.10 to v0.9.11 moonbeam-foundation/moonbeam#892

Closed

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

DatabaseSource::Auto #9500

DatabaseSource::Auto #9500

Uh oh!

debris commented Aug 4, 2021 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

arkpar commented Aug 4, 2021 •

edited

Loading

Uh oh!

Uh oh!

debris commented Aug 6, 2021

Uh oh!

Uh oh!

arkpar Aug 7, 2021

Uh oh!

debris commented Aug 9, 2021

Uh oh!

arkpar commented Aug 9, 2021

Uh oh!

ghost commented Aug 9, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

DatabaseSource::Auto #9500

DatabaseSource::Auto #9500

Uh oh!

Conversation

debris commented Aug 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

arkpar commented Aug 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

debris commented Aug 6, 2021

Uh oh!

Uh oh!

arkpar Aug 7, 2021

Choose a reason for hiding this comment

Uh oh!

debris commented Aug 9, 2021

Uh oh!

arkpar commented Aug 9, 2021

Uh oh!

ghost commented Aug 9, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

debris commented Aug 4, 2021 •

edited

Loading

arkpar commented Aug 4, 2021 •

edited

Loading