Skip to content
This repository was archived by the owner on Jul 4, 2022. It is now read-only.

Conversation

@hoani
Copy link
Contributor

@hoani hoani commented Feb 18, 2020

Upstream Changes

Pull and merge changes up to paritytech/substrate@877e193.

Omitted "Composite Accounts (#4820)":
paritytech/substrate@29454c3
^ We are yet to determine whether the interface changes introduced by this commit are worth bringing into plug.

Changed:

  • A bunch of benchmarking tests were added to Balances, Identity, and Timestamp
  • Major refactors on transaction-factory and sc-executor-wasmtime
  • Existential deposit on Balances must be >= 1.
  • SimpleArithmetic trait replaced with AtLeast32Bit in a few places

Concerns:

  • client/transaction-pool/src/testing/pool.rs Ln 438 fails on Local tests (but pass in CI), probably because it is asserting on asynchronous behaviour
  • prml/doughnut/src/impls.rs Ln148, try_into() no longer worked probably because SimpleArithmetic was replaced with BaseArithmetic (as far as I can tell). Replaced it with into, which works, but it is questionable whether or not this is acceptable.

h4x3rotab and others added 30 commits February 10, 2020 09:26
* Export crypto_full feature in primitives/finality-grandpa

* Export GRANDPA AuthorityPair when full_crypto is enabled
* Starting

* closer

* Compiles!

* comments

* Create seperate mock

* Remove changes to test env

* Fix step calculation

* Add host function

* Add runtime api

* compiles

* Update to use offchain timestamp

* Gives a result

* added some CLI wip

* make generic

* Update instance

* Remove CLI stuff

* Remove last cli stuff

* undo more changes

* Update benchmarks

* Update Cargo.lock

* remove test

* Move loop out of runtime

* Benchmarking externalities

* Benchmarking state

* Implemented commit

* Make CLI work, move loop back into runtime

* Wipe resets to genesis

* Speedup benchmarks

* Use enum to select extrinsic within pallet

* CLI controls which module and extrinsic to call

* Select a pallet with cli

* Add steps and repeats to cli

* Output as CSV format

* Introduce benchmark pallet

* Append bench

* Use Results

* fix merge

* Clear Identity benchmark

* Bench request judgment and cancel request

* Add final benchmarks

* Fix CSV output

* Start cleaning up for PR

* Bump numbers in `wasmtime` integration tests.

* More docs

* Add rockdb feature to bench

* Fix formatting issues

* Add test feature to bench

* Add test feature to bench

* Add rocksdb feature flag

* Update bench.rs

Co-authored-by: Arkadiy Paronyan <[email protected]>
Co-authored-by: Gavin Wood <[email protected]>
* Fix vesting logic

* Bump runtime version

* Docs.
* Make WASM browser thing compile

* Fix

* updated exit-future (github repo)

* Switch to broadcast crate

* Migrate client/cli

* Switch exit-future to modernize branch

* Small changes

* Switch to cargo version and fix fg tests

* fix basic-authorship

* Fix crash on grafana macro

* Fix grafana macro

* Switch node python version

* Disable record_metrics_slice in grafana macro on wasm

* Update client/grafana-data-source/src/lib.rs

* Revert "Update client/grafana-data-source/src/lib.rs"

This reverts commit 888009a8e0b7051bd4bfbbfdb0448bcf2e2aae93.

* Add wasm support for state machine

* Switch to my own libp2p version

* Revert "Switch to my own libp2p version"

This reverts commit ce613871b59264b3165b45c37943e6560240daa7.

* Revert "Add wasm support for state machine"

This reverts commit de7eaa0694d9534fc3b164621737968e9a6a7c5f.

* Add sc-browser

* Squash

* remove sc-browser

* Fix keystore on wasm

* stubs for removed functions to make env compatible with old runtimes

* Add test (that doesn't work)

* Fix build scripts

* Revert basic-authorship due to no panics

* Revert cli/informant

* Revert consensus

* revert offchain

* Update utils/browser/Cargo.toml

Co-Authored-By: Benjamin Kampmann <[email protected]>

* export console functions

* Add new chainspec

* Fix ws in chain spec

* revert chainspec

* Fix chainspec

* Use an Option<PathBuf> in keystore instead of cfg flags

* Remove crud

* Only use wasm-timer for instant and systemtime

* Remove telemetry changes

* Assuming this is ok

* Add a KeystoreConfig

* Add stubs back in

* Update libp2p

* Revert "Add stubs back in"

This reverts commit 4690cf1882aa0f99f7f00a58c4080c8aa9b77c36.

* Remove commented js again

* Bump kvdb-web version

* Fix cli

* Switch branch on futures-timer

* Fix tests

* Remove sc-client test build in check-web-wasm because there isn't a good way to build futures-timer with wasm-bindgen support in the build

* Remove more things ^^

* Switch branch on futures-timer back

* Put DB io stats behind a cfg flag

* Fix things

* Don't timeout transports on wasm

* Update branch of futures-timer and fix bad merge

* Spawn informant

* Fix network test

* Fix delay resets

* Changes

* Fix tests

* use wasm_timer for transaction pool

* Fixes

* Switch futures-timer to crates

* Only diagnose futures on native

* Fix sc-network-test tests

* Select log level in js

* Fix syncing ;^)

* Allow disabling colours in the informant

* Use OutputFormat enum for informant

* MallocSizeOf impl on transaction pool broke stuff because wasm_timer::Instant doesnt impl it so just revert the transaction pool to master

* Update futures-diagnose

* Revert "MallocSizeOf impl on transaction pool broke stuff because wasm_timer::Instant doesnt impl it so just revert the transaction pool to master"

This reverts commit baa4ffc94fd968b6660a2c17ba8113e06af15548.

* Pass whole chain spec in start_client

* Get Instant::now to work in transaction pool again

* Informant dep reordering

Co-authored-by: Pierre Krieger <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Svyatoslav Nikolsky <[email protected]>
Co-authored-by: Benjamin Kampmann <[email protected]>
Co-authored-by: Demi Obenour <[email protected]>
* Don't expose `Benchmarking` host functions by default

* Fix tests

Co-authored-by: Shawn Tabrizi <[email protected]>
* Clarify code a bit.

* Move code around.

* Introduce `Order`.

* Introduce `Link` structure.

* Get rid of ptr_offset

This is beneficial since ptr_offset is essentially makes us handle two different address spaces, global (i.e. `mem`) and heap local and without it things are becoming simpler.

* Rename PREFIX_SIZE to HEADER_SIZE.

This will come in the next commits.

* Introduce a separate `Memory` trait.

This is not necessary, but will come in handy for the upcoming changes.

* Rename `ptr` to `header_ptr` where makes sense.

* Introduce a `Header` type.

* Make `bump` dumber.

This allows us to pull `HEADER_SIZE` to see that we actually allocate `order.size() + HEADER_SIZE`.

* Clean up.

* Introduce a freelists struct.

* Update documentation.

* Make Sized requirement optional to make the PR truly back-compatible.

* Apply suggestions from code review

Co-Authored-By: Gavin Wood <[email protected]>

* Apply suggestions from code review

Co-Authored-By: Bastian Köcher <[email protected]>

Co-authored-by: Gavin Wood <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
* Merge branch 'gav-split-balanecs-vesting' into gav-upsub

# Conflicts:
#	Cargo.lock
#	cli/Cargo.toml
#	collator/Cargo.toml
#	primitives/Cargo.toml
#	runtime/common/Cargo.toml
#	runtime/common/src/claims.rs
#	runtime/kusama/Cargo.toml
#	runtime/polkadot/Cargo.toml
#	service/Cargo.toml

* Fix tests
* Add trait to get module and call names.


Co-authored-by: Bastian Köcher <[email protected]>
* Run offchain workers at particular hash, not number.

* Don't run if not new best.

* Don't run if not new best.

* Update client/service/src/builder.rs

Co-Authored-By: Nikolay Volf <[email protected]>

* Update client/service/src/builder.rs

Co-Authored-By: Nikolay Volf <[email protected]>

* Update client/service/src/builder.rs

Co-authored-by: Nikolay Volf <[email protected]>
* Add support for json output in subkey

* Updates as per code review

* Apply suggestions from code review

Co-Authored-By: Nikolay Volf <[email protected]>

* Apply suggestions from code review

Co-Authored-By: Bastian Köcher <[email protected]>

* Clean up error handler as per code review

* Apply suggestions from code review

Co-Authored-By: Marcio Diaz <[email protected]>

* Fix compilation error

* Remove accidental file commit

Co-authored-by: Nikolay Volf <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Marcio Diaz <[email protected]>
…#4886)

* impl Randomness trait for Babe and remove unused RandomBeacon trait

* bump runtime version
* Pause Kademlia if too many connections

* Fix test

* Update client/network/src/discovery.rs

Co-Authored-By: Toralf Wittner <[email protected]>

* Change the limit

Co-authored-by: Toralf Wittner <[email protected]>
* Remove modes.

* Refactor.
* pallet-evm: optional nonce parameter

* Consume all gases when nonce mismatches

* Bump node runtime version
* Add a sub command to generate a node key file in the format required by a substrate node

* Update lock file

* Apply suggestions from code review

Co-Authored-By: Bastian Köcher <[email protected]>

* Updates as per code review

Co-authored-by: Bastian Köcher <[email protected]>
* Add selected_benchmark! macro.

* Use selected_benchmark! in Identity pallet.

* Implement timestamp pallet benchmark.

* Fix some nits.

* Bump impl_version.
* Add command-line flag to enable yamux flow control.

We never enabled proper flow-control for yamux streams which may cause
stream buffers to exceed their configured limit when the stream
producer outpaces the stream consumer. By switching the window update
mode to on-read, producers will only receive more sending credit when
all data has been consumed from the stream buffer. Using this option
creates backpressure on producers. However depending on the protocol
there is a risk of deadlock, if both endpoints concurrently attempt to
send more data than they have credit for and neither side reads before
finishing their writes. To facilitate proper testing, this PR adds a
command-line flag `use-yamux-flow-control`.

* Replace comment with generic message.
* Add non-zero ed check on Balances genesis

* Update ED from 0 to 1

* bump impl

* bump spec

* Found remove more ed = 0

* Fix some contract tests

* Use ctx.overlay.set_balance for contracts

* Fix staking test

* Remove obsolete logic

* Allow death of payout account in society

* Update frame/balances/src/lib.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Dont create genesis balances if balance is zero in transaction payment pallet

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Gavin Wood <[email protected]>
* network: Use "one shot" protocol handler.

Add two new `NetworkBehaviour`s, one handling remote block requests
and another one to handle light client requests (both local and from
remote). The change is motivated by the desire to use multiple
substreams of a single connection for different protocols. To achieve
this, libp2p's `OneShotHandler` is used as a protocol handler in each
behaviour. It will open a fresh substream for the duration of the
request and close it afterwards. For block requests, we currently only
handle incoming requests from remote and tests are missing. For light
client handling we support incoming requests from remote and also
ported a substantial amount of functionality over from
`light_dispatch.rs` (including several tests). However the result lacks
in at least two aspects:

(1) We require external updates w.r.t. the best block per peer and
currently nothing updates this information.
(2) We carry a lot of peer-related state around.

Both aspects could be simplified by externalising peer selection and
just requiring a specific peer ID where the request should be sent to.
We still have to maintain some peer related state due to the way
libp2p's swarm and network behaviour work (e.g. we must make sure to
always issue `NetworkBehaviourAction::SendEvent`s to peers we are
connected to, otherwise the actions die a silent death.

Another change implemented here is the use of protocol buffers as the
encoding for network messages. Certain individual fields of messages
are still SCALE encoded. There has been some discussion about this
in another PR (paritytech/substrate#3452), so
far without resolution.

* Uncomment `Behaviour::light_client_request`.

* Add license headers.
* Initial transfer bench

* Add best case

* Transfer keep alive

* Set balance benchmarks

* Bump impl

* Fix text

Co-authored-by: Gavin Wood <[email protected]>
…(#4767)

`GossipEngine` spawns two tasks, one for a periodic tick, one to forward
messages from the network to subscribers. These tasks hold an `Arc` to a
`GossipEngineInner`.

To reduce the amount of shared ownership (locking) this patch integrates
the two tasks into a `Future` implementation on the `GossipEngine`
struct. This `Future` implementation can now be called from a single
owner, e.g. the `finality-grandpa` `NetworkBridge`.

As a side effect this removes the requirement on the `network-gossip`
crate to spawn tasks and thereby removes the requirement on the
`finality-grandpa` crate to spawn any tasks.

This is part of a greater effort to reduce the number of owners of
components within `finality-grandpa`, `network` and `network-gossip` as
well as to reduce the amount of unbounded channels. For details see
d4fbb89, f0c1852 and 5afc777.
* serialize partial_fee into string

* implement deserialize

* bump version
bkchr and others added 15 commits February 13, 2020 10:06
* Fix chain-spec and make sure it does not breaks again

* REview feedback
* full block import benchmark

* try rocksdb cache

* add profiling helper

* use random keyring instead of zero caching

* update docs

* add more io stats

* remove last sentence

* add ci job to see

* Update primitives/keyring/src/sr25519.rs

Co-Authored-By: Marcio Diaz <[email protected]>

* switch to 100tx-block

* remove ci script

Co-authored-by: Marcio Diaz <[email protected]>
* Give perthigns the trait it always deserved.

* Make staking and phragmen work with the new generic per_thing

* Make everything work together 🔨

* a bit of cleanup

* Clean usage

* Bump.

* Fix name

* fix grumbles

* hopefully fix the ui test

* Some grumbles

* revamp traits again

* Better naming again.
* Migrate wasmtime backend to wasmtime-api

* Port to a newer version of wasmtime

* Update to the latest changes.

* Rejig the sandbox module a bit

* Materialze

* Fixes.

* executor wasm_runtime fix

* Refactor everything

* More refactoring

* Even more refactorings

* More cleaning.

* Update to the latest wasmtime

* Reformat

* Renames

* Refactoring and comments.

* Docs

* Rename FunctionExecutor to host.

* Imrpove docs.

* fmt

* Remove panic

* Assert the number of arguments are equal between wasmtime and hostfunc.

* Comment a possible panic if there is no corresponding value variant.

* Check signature of the entrypoint.

* Use git version of wasmtime

* Refine and doc the sandbox code.

* Comment RefCells.

* Update wasmtime to the latest-ish master.

This may solve a problem with segfaults.

* Apply suggestions from code review

Co-Authored-By: Tomasz Drwięga <[email protected]>

* Use full SHA1 hash of wasmtime commit.

* Add a panic message.

* Add some documentation

* Update wasmtime version to include SIGSEGV fix

* Update to crates.io version of wasmtime

* Make it work.

* Move the creation of memory into `InstanceWrapper::new`

* Make `InstanceWrapper` !Send & !Sync

* Avoid using `take_mut`

* Update client/executor/wasmtime/Cargo.toml

Co-Authored-By: Bastian Köcher <[email protected]>

* Limit maximum size of memory.

* Rename `init_state` to `with_initialized_state`

Co-authored-by: Tomasz Drwięga <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
* pallet-evm: add support for transaction-level create2

* Bump runtime version

* Switch to FunctionOf for weights
* pallet-evm: refactor duplicate code in call/create/create2

* Bump runtime version
…me (#4903)

* Adds a test to ensure that we clear the heap between calls into runtime

The tests shows that we currently not clearing the heap in wasmtime.
For now we don't run the test for wasmtime.

* Fix compilation
* Basic account composition.

* Add try_mutate_exists

* De-duplicate

* Refactor away the UpdateBalanceOutcome

* Expunge final UpdateBalanceOutcome refs

* Refactor transfer

* Refactor reservable currency stuff.

* Test with the alternative setup.

* Fixes

* Test with both setups.

* Fixes

* Fix

* Fix macros

* Make indices opt-in

* Remove CreationFee, and make indices opt-in.

* Fix construct_runtime

* Fix last few bits

* Fix tests

* Update trait impls

* Don't hardcode the system event

* Make tests build and fix some stuff.

* Pointlessly bump runtime version

* Fix benchmark

* Another fix

* Whitespace

* Make indices module economically safe

* Migrations for indices.

* Fix

* Whilespace

* Trim defunct migrations

* Remove unused storage item

* More contains_key fixes

* Docs.

* Bump runtime

* Remove unneeded code

* Fix test

* Fix test

* Update frame/balances/src/lib.rs

Co-Authored-By: Shawn Tabrizi <[email protected]>

* Fix ED logic

* Repatriate reserved logic

* Typo

* Fix typo

* Update frame/system/src/lib.rs

Co-Authored-By: Shawn Tabrizi <[email protected]>

* Update frame/system/src/lib.rs

Co-Authored-By: Shawn Tabrizi <[email protected]>

* Last few fixes

* Another fix

* Build fix

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Jaco Greeff <[email protected]>
Co-authored-by: Shawn Tabrizi <[email protected]>
* contracts: Allow to distinguish out of gas from other traps

When a contract encounters a runtime error a wasm trap is
triggered and the execution is halted. Currently, no matter
what was the cause for the trap it is always reported as:
DispatchError::Other("contract trapped during execution").

However, the trap that is triggered if a contract exhausts
its gas budget is particulary interesting. Therefore we add
a seperate error message for this cause:
DispatchError::Other("ran out of gas during contract execution").

A test is added hat executes a contract that never terminates.
Therefore it always exhausts is gas budget.

* fixup! contracts: Allow to distinguish out of gas from other traps

Remove overlong lines.

* fixup! contracts: Allow to distinguish out of gas from other traps

Rename Contract to Contracts
…l watchers. (#4740)

* adds finalization support to sc-transaction-pool using MaintainedTransactionPool for finalization events

* adds TransactionStatus::Retracted, notify watchers of retracted blocks, finalized now finalizes, transactions for current finalized -> last finalized block

* adds last_finalized to ChainApi, use generic BlockT for ChainEvent

* fix tests

* Apply suggestions from code review

Co-Authored-By: Tomasz Drwięga <[email protected]>

* tests

* fix tests, docs, lazily dedupe pruned hashes

* fix tests, Cargo.lock

* Apply suggestions from code review

Co-Authored-By: Tomasz Drwięga <[email protected]>

* remove tree_route, last_finalized from ChainApi, add block hash to Finalization and Retracted events

* prune finality watchers

* fix tests

* remove HeaderBackend bound from FullChainApi

* code style nits, terminate stream in finality_timeout

Co-authored-by: Tomasz Drwięga <[email protected]>
* Adds `with_pair!` macro to application-crypto

This macro will "generate" the given code only when the crypto pair is
available. So, when either the `std` or the `full_crypto` feature is
enabled.

* Fix example
* Remove rename for finalized event.

* ADd some docs.

* Document timeout.
@hoani hoani marked this pull request as ready for review February 18, 2020 20:23
Copy link
Contributor

@mzxyz mzxyz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work, LGTM

@hoani
Copy link
Contributor Author

hoani commented Feb 18, 2020

image
Local test failure is repeatable.

Copy link
Contributor

@jordy25519 jordy25519 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Re concerns, Into usage in doughnut looks ok to me
Failing test is upstream problem: Some related temp fix in: https://github.com/paritytech/substrate/pull/4967/files

@jordy25519 jordy25519 merged commit 613e545 into develop Feb 19, 2020
@jordy25519 jordy25519 deleted the chores/merge-upstream branch February 19, 2020 02:12
@jordy25519 jordy25519 restored the chores/merge-upstream branch February 27, 2020 23:12
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.