feat/ocw/bookkeeping #5200

drahnr · 2020-03-10T07:04:40Z

Changes in order to allow writing data during block import to the offchain database, so that the offchain worker has access to this data.

The implementation introduces a new interface accessible from runtime.

Related #3722

polkadot companion: paritytech/polkadot#1025

tomusdrw

Looks good overall, I'd change the logic a bit though.

import_notification_stream is left as it works now - not guaranteed for every block, and also not running for initial sync. Has some extra data in the notification though.
2 all_blocks_import_notification_stream - a more "lightweight" stream, but guaranteed to run for each and every imported block. This is what offchain workers should be using now. Additionally the information if it was an initial sync or not should be passed as parameter in the stream, so that we can for instance keep backward compatibility for extisting offchain workers (the filtering would be done within the runtime though, not outside).

client/api/src/client.rs

client/cli/src/params/offchain_worker_params.rs

client/service/src/builder.rs

client/src/client.rs

drahnr · 2020-03-30T14:11:16Z

Note that this contains changes of #5455

client/cli/src/params/offchain_worker_params.rs

client/service/src/builder.rs

client/src/client.rs

Co-Authored-By: Kian Paimani <[email protected]>

kianenigma

None were critical to request changes, but I had some comments that preferably need addressing before merge.

tomusdrw

A bunch of docs improvements, functionally it looks good.

client/api/src/backend.rs

client/cli/src/config.rs

client/cli/src/params/offchain_worker_params.rs

primitives/io/src/lib.rs

tomusdrw · 2020-04-23T12:50:30Z

primitives/state-machine/src/basic.rs

 }

 impl Externalities for BasicExternalities {
+	fn set_offchain_storage(&mut self, _key: &[u8], _value: Option<&[u8]>) {}


I think in this case it's better to work with borrowed data, cause we can avoid a clone if indexing is disabled. That said we should probably unify the API (and introduce Storage*Ref to work with slices). I'd leave it as a separate PR though.

primitives/state-machine/src/ext.rs

Co-Authored-By: Tomasz Drwięga <[email protected]>

rphmeier · 2020-04-23T22:34:06Z

Companion and this look good, I think we can merge once CI completes here. cc @TriplEight companion PR task may not be working correctly; I've tried restarting it but we'll see if it works.

TriplEight · 2020-04-24T09:56:28Z

@rphmeier I've checked and it seems to be caused by GitHub availability issues.

cheme · 2020-05-04T13:55:53Z

Quick question, should the overlay maintain a commit and prospective state?
(currently rejected extrinsic will index, but that may not be an issue (if reorg is not this is maybe not either).

rphmeier · 2020-05-04T14:14:08Z

Good point - although at the moment our use-cases are in on_initialize/on_finalize only, we should be able to handle rollback better.

drahnr self-assigned this Mar 10, 2020

drahnr force-pushed the bernhard/deterministic-bookkeeping-issue-3722 branch from ed6990d to 5d11ae9 Compare March 10, 2020 07:54

gavofyork added the A3-in_progress Pull request is in progress. No review needed at this stage. label Mar 10, 2020

drahnr requested a review from tomusdrw March 10, 2020 11:00

drahnr force-pushed the bernhard/deterministic-bookkeeping-issue-3722 branch from 075c9f9 to d5ff49b Compare March 11, 2020 10:17

tomusdrw reviewed Mar 11, 2020

View reviewed changes

client/src/client.rs Outdated Show resolved Hide resolved

drahnr force-pushed the bernhard/deterministic-bookkeeping-issue-3722 branch from f4868db to e9ca0eb Compare March 11, 2020 18:23

tomusdrw mentioned this pull request Mar 12, 2020

Produce block always on updated transaction pool state #5227

Merged

drahnr force-pushed the bernhard/deterministic-bookkeeping-issue-3722 branch from f059032 to 28a464c Compare March 16, 2020 13:06

paritytech-ci added the B2-breaksapi label Mar 23, 2020

drahnr force-pushed the bernhard/deterministic-bookkeeping-issue-3722 branch 2 times, most recently from c0dfe8b to 1b76596 Compare March 24, 2020 08:32

drahnr requested a review from tomusdrw March 24, 2020 09:05

drahnr force-pushed the bernhard/deterministic-bookkeeping-issue-3722 branch from 74d1137 to f09046b Compare March 30, 2020 13:51

drahnr linked an issue Mar 30, 2020 that may be closed by this pull request

Offchain Workers: deterministic bookkeeping #3722

Closed

drahnr marked this pull request as ready for review March 30, 2020 14:08

drahnr requested review from NikVolf and pepyakin as code owners March 30, 2020 14:08