Skip to content
This repository was archived by the owner on Nov 15, 2023. It is now read-only.
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
153 commits
Select commit Hold shift + click to select a range
ef20f40
init some bench
cheme Jul 9, 2019
a218162
Init implementing.
cheme Jul 9, 2019
4e8f6a7
Initial implementation of prospective with unbound layer.
cheme Jul 11, 2019
917e26b
Remove size counter
cheme Jul 11, 2019
8b93ab7
fix ix.
cheme Jul 11, 2019
ae8e880
code for transactional layers: TODO testing!!
cheme Jul 12, 2019
c716f95
Regain perfs (-15%) by using a couple of static memory for
cheme Jul 30, 2019
808a409
Fix tx logic and basic tx test.
cheme Jul 30, 2019
e481952
with_transaction function and high level test.
cheme Jul 30, 2019
d9f3e50
state was useless in overlayed change (allways the last ix of history).
cheme Jul 30, 2019
cc0aa37
Cleaning comments and names.
cheme Jul 30, 2019
2bc2af0
Merge branch 'master' into overlay_change
cheme Jul 30, 2019
0c8235e
Rename Externalities primitives to 'storage_' + old name.
cheme Jul 31, 2019
5b98b4b
Merge branch 'master' into overlay_change
cheme Jul 31, 2019
cade913
name fields instead of tuble.
cheme Jul 31, 2019
fd26b2a
Update srml/support/src/storage/mod.rs
cheme Jul 31, 2019
3245968
Use more efficient smallvec.
cheme Aug 1, 2019
8550ad9
Merge branch 'overlay_change' of github.com:cheme/polkadot into overl…
cheme Aug 1, 2019
41483b5
Missing spaces.
cheme Aug 1, 2019
44f5173
slice and slice to_vec from ptr to vec.
cheme Aug 1, 2019
00326b2
Init some fuzzing (only look for panic).
cheme Aug 1, 2019
fc2100d
Implement simplier inefficient overlay to fuzz compare.
cheme Aug 1, 2019
e38fc7b
Update core/state-machine/src/overlayed_changes.rs
cheme Aug 1, 2019
b8564e6
implement overlay garbage collection.
cheme Aug 5, 2019
21aa65d
Trigger gc based on constant rule.
cheme Aug 5, 2019
c8938d3
Managing a configuration for gc of overlay. This commit will be revert
cheme Aug 5, 2019
d1d947b
Revert "Managing a configuration for gc of overlay. This commit will …
cheme Aug 5, 2019
f4fb1e1
Merge branch 'master' into overlay_change
cheme Aug 6, 2019
2836808
Merge branch 'master' into overlay_change
cheme Aug 8, 2019
968ba35
Merge branch 'master' into overlay_change
cheme Aug 8, 2019
d703ee4
Merge branch 'master' into overlay_change
cheme Aug 15, 2019
ee52917
Direct iterator on children, not that the fact that it is owned does not
cheme Aug 15, 2019
249675e
Merge branch 'overlay_change' of github.com:cheme/polkadot into overl…
cheme Aug 15, 2019
0100d70
Merge branch 'master' into overlay_change
cheme Aug 22, 2019
da093d5
Merge branch 'master' into overlay_change
cheme Aug 26, 2019
af4cceb
warning fix
cheme Aug 26, 2019
4e68123
Extracting transaction history and states plus logics to its own
cheme Aug 26, 2019
75fd798
Make history buffer parametric (to allow encoding efficient
cheme Aug 26, 2019
1fb154e
Removed history trait, to go to the more pragmatic macro approach (after
cheme Aug 27, 2019
46de531
macroed code (trait would be better but not at this point).
cheme Aug 28, 2019
80f4c2b
Fix basis code.
cheme Aug 28, 2019
8dbd48d
Remove serialized variant.
cheme Aug 28, 2019
9203e25
change some variables name, try to document gc.
cheme Aug 28, 2019
ea028ec
Change operation cost proto, add into_pending method and few formatting
cheme Aug 28, 2019
c69e258
Merge branch 'master' into overlay_change
cheme Aug 28, 2019
0ab789e
Doc and renaming 'force_push' to 'unsafe_push' (this function being
cheme Aug 29, 2019
8d45d31
Merge branch 'master' into overlay_change
cheme Aug 29, 2019
9488ca1
Merge branch 'master' into overlay_change
cheme Sep 2, 2019
09a24e0
Merge branch 'overlay_change' of github.com:cheme/substrate into over…
cheme Sep 2, 2019
b596b52
Merge branch 'master' into overlay_change
cheme Sep 4, 2019
5cc5fd0
Merge branch 'master' into overlay_change
cheme Sep 5, 2019
117388e
bump spec version
cheme Sep 5, 2019
1647084
Merge branch 'master' into overlay_change
cheme Sep 11, 2019
838cd3b
Addressing some nits, adding fuzz check into ci benches target.
cheme Sep 12, 2019
5437981
Merge branch 'master' into overlay_change
cheme Sep 12, 2019
fe38b30
Update iter docs, bump runtime version.
cheme Sep 13, 2019
b7dcd6d
Merge branch 'master' into overlay_change
cheme Sep 13, 2019
616fc2a
Simplify top and child iter into a single function.
cheme Sep 13, 2019
7001e8c
Restore access to iter for fuzzer.
cheme Sep 13, 2019
b534b3d
Merge branch 'master' into overlay_change
cheme Sep 16, 2019
a84b8db
rename gc for liner to get_mut_pruning, rename unsafe_push to
cheme Sep 16, 2019
9787a97
switch tuple to historiedvalue struct.
cheme Sep 16, 2019
d9d2118
use len instead of internal_item_counts
cheme Sep 16, 2019
887f95b
historied value on get and get mut to.
cheme Sep 17, 2019
1074c8c
Merge branch 'master' into overlay_change
Demi-Marie Sep 17, 2019
076623d
Merge branch 'master' into overlay_change
cheme Sep 18, 2019
937655f
Merge branch 'overlay_change' of github.com:cheme/polkadot into overl…
cheme Sep 18, 2019
ff4bcb0
proper rand on bench
cheme Sep 18, 2019
45dc935
Update core/utils/historied-data/README.md
cheme Sep 20, 2019
d7c9f8a
Update core/utils/historied-data/src/lib.rs
cheme Sep 20, 2019
6c9abda
Update core/utils/historied-data/src/linear.rs
cheme Sep 20, 2019
89b4f26
Switching runtime test to panic instead of returning result.
cheme Sep 20, 2019
2ab0da3
Merge branch 'overlay_change' of github.com:cheme/polkadot into overl…
cheme Sep 20, 2019
6673ce1
Merge branch 'master' into overlay_change
cheme Oct 2, 2019
cdd1c3e
Merge branch 'master' into overlay_change
cheme Oct 10, 2019
e076d12
Avoid clone explicitely in some case (the clone with this PR is
cheme Oct 10, 2019
b877213
remove it from parameter.
cheme Oct 10, 2019
fc66ecd
Merge branch 'master' into overlay_change
cheme Oct 18, 2019
dcdd32c
Merge branch 'master' into overlay_change
cheme Oct 21, 2019
6a153e5
Merge branch 'overlay_change' of github.com:cheme/polkadot into overl…
cheme Oct 21, 2019
32e6268
fix merge.
cheme Oct 21, 2019
cf943e2
Apply suggestions from code review
cheme Oct 21, 2019
4a439af
Merge branch 'overlay_change' of github.com:cheme/polkadot into overl…
cheme Oct 21, 2019
c71e2c5
Smallvec is compatible with no_std.
cheme Oct 21, 2019
71c4012
fix fuzz, rename 'test' feature to more substratish 'test-helpers'.
cheme Oct 21, 2019
4ca0b40
Implement with less state.
cheme Oct 21, 2019
47b8867
revert to no gc of commited data
cheme Oct 23, 2019
d9a0fb5
renaming some history variables and field to states.
cheme Oct 23, 2019
1c06ce1
satisfactory get_mut, pruning need changes.
cheme Oct 23, 2019
c399c58
Fix gc (fuzz seems ok), a bit more work on avoiding redundancy in
cheme Oct 23, 2019
106d84b
remove a bit of code redundancy.
cheme Oct 24, 2019
6b5aeea
rename historied to historical
cheme Oct 24, 2019
8738b30
Merge branch 'master' into overlay_change
cheme Oct 24, 2019
594ce64
removing dubious as_ref_mut method (intermediate tuple is not optimized
cheme Oct 24, 2019
7546e6a
Named field for States.
cheme Oct 24, 2019
2e6fa4c
change find_previous_transaction to use an in memory state.
cheme Oct 24, 2019
cd7113b
update runtime versions.
cheme Oct 24, 2019
a995150
Merge branch 'master' into overlay_change
cheme Oct 24, 2019
789af78
Merge branch 'master' into overlay_change, and few english nits.
cheme Oct 28, 2019
1a12715
Merge branch 'master' into overlay_change
cheme Oct 30, 2019
6625ab8
Merge branch 'overlay_change' of github.com:cheme/polkadot into overl…
cheme Oct 30, 2019
2d91b89
fix some documentation nits.
cheme Oct 30, 2019
61dba1e
Documentation.
cheme Nov 1, 2019
6334c10
favor for expression when possible.
cheme Nov 1, 2019
1b9ff7b
use 'get_unchecked_pending' for a recurrent pattern matching.
cheme Nov 1, 2019
55cacc1
Reorganize historical crate and documentation.
cheme Nov 1, 2019
4920da4
fix import.
cheme Nov 1, 2019
4a1fb7f
Preparing refactor.
cheme Nov 11, 2019
a995e18
setting up linear synch historical data.
cheme Nov 12, 2019
fb01856
quick implementation before fuzzing.
cheme Nov 12, 2019
ef0bcda
code fix and overlay implemetation, before deletion of old code.
cheme Nov 13, 2019
e265ad1
New implementation accounts for a 17 and 20% lost on existing bench.
cheme Nov 13, 2019
9904c4f
Remove unuse code. Test failings.
cheme Nov 14, 2019
0a98033
Merge branch 'master' into overlay_change
cheme Nov 14, 2019
5f82305
Fix broken tests (obviously cloning only state do not work anymore).
cheme Nov 14, 2019
494ad42
Removing trailing spaces.
cheme Nov 14, 2019
f7b9594
rename 'ensure_runing' to 'finalize_discard' as it is only really needed
cheme Nov 14, 2019
55d112a
renaming PruneResult to CleaningResult.
cheme Nov 14, 2019
741d420
fix compilation issue
cheme Nov 14, 2019
19b1014
Field name for states.
cheme Nov 15, 2019
a2bab20
Merge branch 'master' into overlay_change
cheme Nov 15, 2019
4b90775
Fix conflicts.
cheme Nov 15, 2019
a81fb12
fix ci for checking fuzz
cheme Nov 15, 2019
d9ff813
No sense in having new ext in deprecated interface.
cheme Nov 19, 2019
fa0ec93
Ensure discard prospective.
cheme Nov 19, 2019
e0b580e
Merge branch 'master' into overlay_change
cheme Nov 21, 2019
efa05f1
Merge branch 'master' into overlay_change
cheme Nov 23, 2019
e0a6243
Merge branch 'master' into overlay_change
cheme Nov 27, 2019
281344b
Merge branch 'master' into overlay_change
cheme Nov 29, 2019
2676397
Merge branch 'master' into overlay_change
cheme Nov 29, 2019
fbdbd34
Merge branch 'master' into overlay_change
cheme Dec 2, 2019
80c55c6
Merge branch 'overlay_change' of github.com:cheme/substrate into over…
cheme Dec 2, 2019
5597bf3
update cargo.lock
cheme Dec 2, 2019
9c09b78
fix crate names for bench and fuzz
cheme Dec 6, 2019
28413a7
Merge branch 'master' into overlay_change
cheme Dec 8, 2019
1e1e790
Address nits: remove test-helpers specific derive, make 'len' function
cheme Dec 9, 2019
827441c
Merge branch 'master' into overlay_change
cheme Dec 9, 2019
ac424ac
Merge branch 'master' into overlay_change
cheme Dec 10, 2019
6d3e10f
Merge branch 'master' into overlay_change
cheme Dec 11, 2019
19733c5
fix merge (runtime-io to sp-io)
cheme Dec 11, 2019
f7a1193
Merge branch 'master' into overlay_change
cheme Dec 11, 2019
0f8bee6
Move historical-data crate to primitives.
cheme Dec 11, 2019
f8cdd1c
Merge branch 'overlay_change' of https://github.com/cheme/substrate i…
pepyakin Dec 12, 2019
fc5121d
Add sp-historical-data to pallet-contracts
pepyakin Dec 16, 2019
aaeb6c7
Add a simple direct storage access module
pepyakin Dec 16, 2019
12986a0
WIP
pepyakin Dec 17, 2019
90328ac
Completely migrate to the transactional system.
pepyakin Dec 18, 2019
ba048c1
Format
pepyakin Dec 18, 2019
ee98efb
Fix wasm compilation
pepyakin Dec 18, 2019
60a8989
Get rid of account_db module
pepyakin Dec 18, 2019
cafe04d
Make deposit event eager
pepyakin Dec 19, 2019
20d8482
Make restore_to eager
pepyakin Dec 19, 2019
55a5a59
Fix it for the non-std builds.
pepyakin Dec 20, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Managing a configuration for gc of overlay. This commit will be revert
(static value makes more sense for the time being).
  • Loading branch information
cheme committed Aug 5, 2019
commit c8938d3cf003a51d55ef3db54f5f20171db2e562
4 changes: 4 additions & 0 deletions core/client/src/backend.rs
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ use sr_primitives::{generic::BlockId, Justification, StorageOverlay, ChildrenSto
use sr_primitives::traits::{Block as BlockT, NumberFor};
use state_machine::backend::Backend as StateBackend;
use state_machine::ChangesTrieStorage as StateChangesTrieStorage;
use state_machine::OverlayedSettings;
use consensus::well_known_cache_keys;
use hash_db::Hasher;
use trie::MemoryDB;
Expand Down Expand Up @@ -196,6 +197,9 @@ pub trait Backend<Block, H>: AuxStore + Send + Sync where
/// something that the import of a block would interfere with, e.g. importing
/// a new block or calculating the best head.
fn get_import_lock(&self) -> &Mutex<()>;

/// Settings to use with overlay.
fn overlay_settings(&self) -> OverlayedSettings;
}

/// Offchain workers local storage.
Expand Down
1 change: 1 addition & 0 deletions core/client/src/call_executor.rs
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,7 @@ where
side_effects_handler: Option<&mut O>,
) -> error::Result<Vec<u8>> {
let mut changes = OverlayedChanges::default();
changes.change_settings(self.backend.overlay_settings());
let state = self.backend.state_at(*id)?;
let return_data = state_machine::new(
&state,
Expand Down
1 change: 1 addition & 0 deletions core/state-machine/src/changes_trie/build.rs
Original file line number Diff line number Diff line change
Expand Up @@ -259,6 +259,7 @@ mod test {
])),
].into_iter().collect(),
},
settings: Default::default(),
operation_from_last_gc: 0,
};

Expand Down
1 change: 1 addition & 0 deletions core/state-machine/src/ext.rs
Original file line number Diff line number Diff line change
Expand Up @@ -380,6 +380,7 @@ mod tests {
])),
].into_iter().collect(),
},
settings: Default::default(),
operation_from_last_gc: 0,
}
}
Expand Down
2 changes: 1 addition & 1 deletion core/state-machine/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ pub use changes_trie::{
prune as prune_changes_tries,
oldest_non_pruned_trie as oldest_non_pruned_changes_trie
};
pub use overlayed_changes::OverlayedChanges;
pub use overlayed_changes::{OverlayedChanges, OverlayedSettings};
pub use proving_backend::{
create_proof_check_backend, create_proof_check_backend_storage,
Recorder as ProofRecorder, ProvingBackend,
Expand Down
64 changes: 46 additions & 18 deletions core/state-machine/src/overlayed_changes.rs
Original file line number Diff line number Diff line change
Expand Up @@ -45,24 +45,42 @@ pub(crate) enum TransactionState {
}


/// Treshold of operation before running a garbage colletion
/// on a transaction operation.
/// Should be same as `TRIGGER_COMMIT_GC` or higher
/// (we most likely do not want lower as transaction are
/// possibly more frequent than commit).
const TRIGGER_TRANSACTION_GC: usize = 10_000;

/// Treshold of operation before running a garbage colletion
/// on a commit operation.
/// We may want a lower value than for a transaction, even
/// a 1 if we want to do it between every operation.
const TRIGGER_COMMIT_GC: usize = 1_000;

/// Used to count big content as multiple operation.
/// This is a number of octet.
/// Set to 0 to ignore.
const ADD_CONTENT_SIZE_UNIT: usize = 64;

#[derive(Debug, Clone)]
/// Settings regarding overlayed changes internal (mainly gc).
pub struct OverlayedSettings {
/// Treshold of operation before running a garbage colletion
/// on a transaction operation.
/// Should be same as `TRIGGER_COMMIT_GC` or higher
/// (we most likely do not want lower as transaction are
/// possibly more frequent than commit).
pub trigger_transaction_gc: usize,
/// Treshold of operation before running a garbage colletion
/// on a commit operation.
/// We may want a lower value than for a transaction, even
/// a 1 if we want to do it between every operation.
pub trigger_commit_gc: usize,
/// Used to count big content as multiple operation.
/// This is a number of octet.
/// Set to 0 to ignore.
pub add_content_size_unit: usize,
}

impl Default for OverlayedSettings {
fn default() -> Self {
OverlayedSettings {
trigger_transaction_gc: TRIGGER_TRANSACTION_GC,
trigger_commit_gc: TRIGGER_COMMIT_GC,
add_content_size_unit: ADD_CONTENT_SIZE_UNIT,
}
}
}

/// The overlayed changes to state to be queried on top of the backend.
///
/// A transaction shares all prospective changes within an inner overlay
Expand All @@ -74,6 +92,8 @@ pub struct OverlayedChanges {
/// Changes trie configuration. None by default, but could be installed by the
/// runtime if it supports change tries.
pub(crate) changes_trie_config: Option<ChangesTrieConfig>,
/// Settings regarding overlayed changes internal (mainly gc).
pub(crate) settings: OverlayedSettings,
/// Counter of number of operation between garbage collection.
/// Add or delete cost one, additional cost per size by counting a fix size
/// as a unit.
Expand Down Expand Up @@ -652,6 +672,13 @@ impl OverlayedChangeSet {
}

impl OverlayedChanges {

/// Change overlayed change settings.
pub fn change_settings(&mut self, settings: OverlayedSettings) {
self.settings = settings;
}


/// Whether the overlayed changes are empty.
pub fn is_empty(&self) -> bool {
self.changes.is_empty()
Expand Down Expand Up @@ -707,8 +734,8 @@ impl OverlayedChanges {
}

fn add_cost_op(&mut self, val: &Option<Vec<u8>>) {
let content_cost = if ADD_CONTENT_SIZE_UNIT > 0 {
val.as_ref().map(|s| s.len() / ADD_CONTENT_SIZE_UNIT).unwrap_or(0)
let content_cost = if self.settings.add_content_size_unit > 0 {
val.as_ref().map(|s| s.len() / self.settings.add_content_size_unit).unwrap_or(0)
} else { 0 };
self.operation_from_last_gc += 1 + content_cost;
}
Expand Down Expand Up @@ -772,7 +799,7 @@ impl OverlayedChanges {
/// Discard prospective changes to state.
pub fn discard_prospective(&mut self) {
self.changes.discard_prospective();
if self.operation_from_last_gc > TRIGGER_COMMIT_GC {
if self.operation_from_last_gc > self.settings.trigger_commit_gc {
self.operation_from_last_gc = 0;
self.gc(true);
}
Expand All @@ -781,7 +808,7 @@ impl OverlayedChanges {
/// Commit prospective changes to state.
pub fn commit_prospective(&mut self) {
self.changes.commit_prospective();
if self.operation_from_last_gc > TRIGGER_COMMIT_GC {
if self.operation_from_last_gc > self.settings.trigger_commit_gc {
self.operation_from_last_gc = 0;
self.gc(true);
}
Expand All @@ -790,7 +817,7 @@ impl OverlayedChanges {
/// Create a new transactional layer.
pub fn start_transaction(&mut self) {
self.changes.start_transaction();
if self.operation_from_last_gc > TRIGGER_TRANSACTION_GC {
if self.operation_from_last_gc > self.settings.trigger_transaction_gc {
self.operation_from_last_gc = 0;
self.gc(true);
}
Expand All @@ -800,7 +827,7 @@ impl OverlayedChanges {
/// A transaction is always running (history always end with pending).
pub fn discard_transaction(&mut self) {
self.changes.discard_transaction();
if self.operation_from_last_gc > TRIGGER_TRANSACTION_GC {
if self.operation_from_last_gc > self.settings.trigger_transaction_gc {
self.operation_from_last_gc = 0;
self.gc(true);
}
Expand Down Expand Up @@ -859,6 +886,7 @@ impl OverlayedChanges {
let mut result = OverlayedChanges {
changes,
changes_trie_config,
settings: Default::default(),
operation_from_last_gc: 0,
};
committed.into_iter().for_each(|(k, v)| result.set_storage(k, v));
Expand Down