Merge remote-tracking branch 'origin/master' into na-jsonrpsee-core-c…

…lient

CHANGELOG.md

@@ -6,6 +6,75 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.16.0] - 2022-02-01
+
+*Note*: This is a significant release which introduces support for V14 metadata and macro based codegen, as well as making many breaking changes to the API.
+
+### Changed
+
+- Log debug message for JSON-RPC response ([#415](https://github.com/paritytech/subxt/pull/415))
+- Only convert struct names to camel case for Call variant structs ([#412](https://github.com/paritytech/subxt/pull/412))
+- Parameterize AccountData ([#409](https://github.com/paritytech/subxt/pull/409))
+- Allow decoding Events containing BitVecs ([#408](https://github.com/paritytech/subxt/pull/408))
+- Custom derive for cli ([#407](https://github.com/paritytech/subxt/pull/407))
+- make storage-n-map fields public too ([#404](https://github.com/paritytech/subxt/pull/404))
+- add constants api to codegen ([#402](https://github.com/paritytech/subxt/pull/402))
+- Expose transaction::TransactionProgress as public ([#401](https://github.com/paritytech/subxt/pull/401))
+- add interbtc-clients to real world usage section ([#397](https://github.com/paritytech/subxt/pull/397))
+- Make own version of RuntimeVersion to avoid mismatches ([#395](https://github.com/paritytech/subxt/pull/395))
+- Use the generated DispatchError instead of the hardcoded Substrate one ([#394](https://github.com/paritytech/subxt/pull/394))
+- Remove bounds on Config trait that aren't strictly necessary ([#389](https://github.com/paritytech/subxt/pull/389))
+- add crunch to readme ([#388](https://github.com/paritytech/subxt/pull/388))
+- fix remote example ([#386](https://github.com/paritytech/subxt/pull/386))
+- fetch system chain, name and version ([#385](https://github.com/paritytech/subxt/pull/385))
+- Fix compact event field decoding ([#384](https://github.com/paritytech/subxt/pull/384))
+- fix: use index override when decoding enums in events ([#382](https://github.com/paritytech/subxt/pull/382))
+- Update to jsonrpsee 0.7 and impl Stream on TransactionProgress ([#380](https://github.com/paritytech/subxt/pull/380))
+- Add links to projects using subxt ([#376](https://github.com/paritytech/subxt/pull/376))
+- Use released substrate dependencies ([#375](https://github.com/paritytech/subxt/pull/375))
+- Configurable Config and Extra types ([#373](https://github.com/paritytech/subxt/pull/373))
+- Implement pre_dispatch for SignedExtensions ([#370](https://github.com/paritytech/subxt/pull/370))
+- Export TransactionEvents ([#363](https://github.com/paritytech/subxt/pull/363))
+- Rebuild test-runtime if substrate binary is updated ([#362](https://github.com/paritytech/subxt/pull/362))
+- Expand the subscribe_and_watch example ([#361](https://github.com/paritytech/subxt/pull/361))
+- Add TooManyConsumers variant to track latest sp-runtime addition ([#360](https://github.com/paritytech/subxt/pull/360))
+- Implement new API for sign_and_submit_then_watch ([#354](https://github.com/paritytech/subxt/pull/354))
+- Simpler dependencies ([#353](https://github.com/paritytech/subxt/pull/353))
+- Refactor type generation, remove code duplication ([#352](https://github.com/paritytech/subxt/pull/352))
+- Make system properties an arbitrary JSON object, plus CI fixes ([#349](https://github.com/paritytech/subxt/pull/349))
+- Fix a couple of CI niggles ([#344](https://github.com/paritytech/subxt/pull/344))
+- Add timestamp pallet test ([#340](https://github.com/paritytech/subxt/pull/340))
+- Add nightly CI check against latest substrate. ([#335](https://github.com/paritytech/subxt/pull/335))
+- Ensure metadata is in sync with running node during tests ([#333](https://github.com/paritytech/subxt/pull/333))
+- Update to jsonrpsee 0.5.1 ([#332](https://github.com/paritytech/subxt/pull/332))
+- Update substrate and hardcoded default ChargeAssetTxPayment extension ([#330](https://github.com/paritytech/subxt/pull/330))
+- codegen: fix compact unnamed fields ([#327](https://github.com/paritytech/subxt/pull/327))
+- Check docs and run clippy on PRs ([#326](https://github.com/paritytech/subxt/pull/326))
+- Additional parameters for SignedExtra ([#322](https://github.com/paritytech/subxt/pull/322))
+- fix: also processess initialize and finalize events in event subscription ([#321](https://github.com/paritytech/subxt/pull/321))
+- Release initial versions of subxt-codegen and subxt-cli ([#320](https://github.com/paritytech/subxt/pull/320))
+- Add some basic usage docs to README. ([#319](https://github.com/paritytech/subxt/pull/319))
+- Update jsonrpsee ([#317](https://github.com/paritytech/subxt/pull/317))
+- Add missing cargo metadata fields for new crates ([#311](https://github.com/paritytech/subxt/pull/311))
+- fix: keep processing a block's events after encountering a dispatch error ([#310](https://github.com/paritytech/subxt/pull/310))
+- Codegen: enum variant indices ([#308](https://github.com/paritytech/subxt/pull/308))
+- fix extrinsics retracted ([#307](https://github.com/paritytech/subxt/pull/307))
+- Add utility pallet tests ([#300](https://github.com/paritytech/subxt/pull/300))
+- fix metadata constants ([#299](https://github.com/paritytech/subxt/pull/299))
+- Generate runtime API from metadata ([#294](https://github.com/paritytech/subxt/pull/294))
+- Add NextKeys and QueuedKeys for session module ([#291](https://github.com/paritytech/subxt/pull/291))
+- deps: update jsonrpsee 0.3.0 ([#289](https://github.com/paritytech/subxt/pull/289))
+- deps: update jsonrpsee 0.2.0 ([#285](https://github.com/paritytech/subxt/pull/285))
+- deps: Reorg the order of deps ([#284](https://github.com/paritytech/subxt/pull/284))
+- Expose the rpc client in Client ([#267](https://github.com/paritytech/subxt/pull/267))
+- update jsonrpsee to 0.2.0-alpha.6 ([#266](https://github.com/paritytech/subxt/pull/266))
+- Remove funty pin, upgrade codec ([#265](https://github.com/paritytech/subxt/pull/265))
+- Use async-trait ([#264](https://github.com/paritytech/subxt/pull/264))
+- [jsonrpsee http client]: support tokio1 & tokio02. ([#263](https://github.com/paritytech/subxt/pull/263))
+- impl `From<Arc<WsClient>>` and `From<Arc<HttpClient>>` ([#257](https://github.com/paritytech/subxt/pull/257))
+- update jsonrpsee ([#251](https://github.com/paritytech/subxt/pull/251))
+- return none if subscription returns early ([#250](https://github.com/paritytech/subxt/pull/250))
+
 ## [0.15.0] - 2021-03-15
 
 ### Added

Cargo.toml

@@ -1,52 +1,12 @@
 [workspace]
-members = [".", "cli", "codegen", "macro", "test-runtime", "client"]
 
-[package]
-name = "subxt"
-version = "0.15.0"
-authors = ["Parity Technologies <[email protected]>"]
-edition = "2021"
-
-license = "GPL-3.0"
-readme = "README.md"
-repository = "https://github.com/paritytech/subxt"
-documentation = "https://docs.rs/subxt"
-homepage = "https://www.parity.io/"
-description = "Submit extrinsics (transactions) to a substrate node via RPC"
-keywords = ["parity", "substrate", "blockchain"]
-include = ["Cargo.toml", "src/**/*.rs", "README.md", "LICENSE"]
-
-[dependencies]
-async-trait = "0.1.49"
-bitvec = { version = "0.20.1", default-features = false, features = ["alloc"] }
-codec = { package = "parity-scale-codec", version = "2", default-features = false, features = ["derive", "full", "bit-vec"] }
-chameleon = "0.1.0"
-scale-info = { version = "1.0.0", features = ["bit-vec"] }
-futures = "0.3.13"
-hex = "0.4.3"
-jsonrpsee = { version = "0.7.0", features = ["macros", "async-client", "client-ws-transport"] }
-log = "0.4.14"
-num-traits = { version = "0.2.14", default-features = false }
-serde = { version = "1.0.124", features = ["derive"] }
-serde_json = "1.0.64"
-thiserror = "1.0.24"
-
-subxt-macro = { version = "0.1.0", path = "macro" }
-
-sp-core = { git = "https://github.com/paritytech/substrate/", branch = "master", default-features = false  }
-sp-runtime = { git = "https://github.com/paritytech/substrate/", branch = "master", default-features = false }
-sp-version = { package = "sp-version", git = "https://github.com/paritytech/substrate/", branch = "master" }
-
-frame-metadata = "14.0.0"
-
-[dev-dependencies]
-sp-arithmetic = { git = "https://github.com/paritytech/substrate/", branch = "master", default-features = false }
-assert_matches = "1.5.0"
-async-std = { version = "1.9.0", features = ["attributes", "tokio1"] }
-env_logger = "0.8.3"
-tempdir = "0.3.7"
-wabt = "0.10.0"
-which = "4.0.2"
-test-runtime = { path = "test-runtime" }
-
-sp-keyring = { package = "sp-keyring", git = "https://github.com/paritytech/substrate/", branch = "master" }
+members = [
+    "cli",
+    "codegen",
+    "examples",
+    "macro",
+    "subxt",
+    "test-runtime",
+    # TODO(niklasad1): remove to separate repo
+    "client"
+]

FILE_TEMPLATE

@@ -1,4 +1,4 @@
-// Copyright 2019-2021 Parity Technologies (UK) Ltd.
+// Copyright 2019-2022 Parity Technologies (UK) Ltd.
 // This file is part of subxt.
 //
 // subxt is free software: you can redistribute it and/or modify

README.md

@@ -8,6 +8,8 @@ A library to **sub**mit e**xt**rinsics to a [substrate](https://github.com/parit
 
 ## Usage
 
+Take a look in the [examples](./examples/examples) folder for various `subxt` usage examples.
+
 ### Downloading metadata from a Substrate node
 
 Use the [`subxt-cli`](./cli) tool to download the metadata for your target runtime from a node.
@@ -26,28 +28,28 @@ a different node then the `metadata` command accepts a `--url` argument.
 
 ### Generating the runtime API from the downloaded metadata
 
-Declare a module and decorate it with the `subxt` attribute which points at the downloaded metadata for the 
+Declare a module and decorate it with the `subxt` attribute which points at the downloaded metadata for the
 target runtime:
 
 ```rust
 #[subxt::subxt(runtime_metadata_path = "metadata.scale")]
 pub mod node_runtime { }
 ```
 
-**Important:** `runtime_metadata_path` resolves to a path relative to the directory where your crate's `Cargo.toml` 
+**Important:** `runtime_metadata_path` resolves to a path relative to the directory where your crate's `Cargo.toml`
 resides ([`CARGO_MANIFEST_DIR`](https://doc.rust-lang.org/cargo/reference/environment-variables.html)), *not* relative to the source file.
 
 ### Initializing the API client
 
-API is still a work in progress. See [examples](./examples) for the current usage.
+API is still a work in progress. See [examples](./examples/examples) for the current usage.
 
 ### Querying Storage
 
-API is still a work in progress. See [tests](./tests/integration/frame) for the current usage.
+API is still a work in progress. See [tests](./subxt/tests/integration/frame) for the current usage.
 
 ### Submitting Extrinsics
 
-API is still a work in progress. See [examples](./examples/polkadot_balance_transfer.rs) for the current usage.
+API is still a work in progress. See [examples](./examples/examples/polkadot_balance_transfer.rs) for the current usage.
 
 ## Integration Testing
 
@@ -67,6 +69,8 @@ Please add your project to this list via a PR.
 - [cargo-contract](https://github.com/paritytech/cargo-contract/pull/79) CLI for interacting with Wasm smart contracts.
 - [xcm-cli](https://github.com/ascjones/xcm-cli) CLI for submitting XCM messages.
 - [phala-pherry](https://github.com/Phala-Network/phala-blockchain/tree/master/standalone/pherry) The relayer between Phala blockchain and the off-chain Secure workers.
+- [crunch](https://github.com/turboflakes/crunch) CLI to claim staking rewards in batch every Era or X hours for substrate-based chains.
+- [interbtc-clients](https://github.com/interlay/interbtc-clients) Client implementations for the interBTC parachain; notably the Vault / Relayer and Oracle.
 
 **Alternatives**
 

RELEASING.md

@@ -0,0 +1,76 @@
+# Release Checklist
+
+These steps assume that you've checked out the Subxt repository and are in the root directory of it.
+
+We also assume that ongoing work done is being merged directly to the `master` branch.
+
+1.  Ensure that everything you'd like to see released is on the `master` branch.
+
+2.  Create a release branch off `master`, for example `release-v0.17.0`. Decide how far the version needs to be bumped based
+    on the changes to date. If unsure what to bump the version to (e.g. is it a major, minor or patch release), check with the
+    Parity Tools team.
+
+3.  Check that you're happy with the current documentation.
+
+    ```
+    cargo doc --open --all-features
+    ```
+
+    CI checks for broken internal links at the moment. Optionally you can also confirm that any external links
+    are still valid like so:
+
+    ```
+    cargo install cargo-deadlinks
+    cargo deadlinks --check-http -- --all-features
+    ```
+
+    If there are minor issues with the documentation, they can be fixed in the release branch.
+
+4.  Bump the crate version in `Cargo.toml` to whatever was decided in step 2 for `subxt-codegen`, `subxt-macro`, `subxt` and `subxt-cli`.
+
+5.  Update `CHANGELOG.md` to reflect the difference between this release and the last. If you're unsure of
+    what to add, check with the Tools team. See the `CHANGELOG.md` file for details of the format it follows.
+
+    Any [closed PRs](https://github.com/paritytech/subxt/pulls?q=is%3Apr+is%3Aclosed) between the last release and
+    this release branch should be noted.
+
+6.  Commit any of the above changes to the release branch and open a PR in GitHub with a base of `master`.
+
+7.  Once the branch has been reviewed and passes CI, merge it.
+
+8.  Now, we're ready to publish the release to crates.io.
+
+    Checkout `master`, ensuring we're looking at that latest merge (`git pull`).
+
+    The crates in this repository need publishing in a specific order, since they depend on each other.
+    Additionally, `subxt-macro` has a circular dev dependency on `subxt`, so we use `cargo hack` to remove
+    dev dependencies (and `--allow-dirty` to ignore the git changes as a result) to publish it.
+
+    So, first install `cargo hack` with `cargo install cargo hack`. Next, you can run something like the following
+    command to publish each crate in the required order (allowing a little time inbetween each to let `crates.io` catch up)
+    with what we've published).
+
+    ```
+    (cd codegen && cargo publish) && \
+        sleep 10 && \
+        (cd macro && cargo hack publish --no-dev-deps --allow-dirty) && \
+        sleep 10 && \
+        (cd subxt && cargo publish) && \
+        sleep 10 && \
+        (cd cli && cargo publish);
+    ```
+
+    If you run into any issues regarding crates not being able to find suitable versions of other `subxt-*` crates,
+    you may just need to wait a little longer and then run the remaining portion of that command.
+
+9.  If the release was successful, tag the commit that we released in the `master` branch with the
+    version that we just released, for example:
+
+    ```
+    git tag -s v0.17.0 # use the version number you've just published to crates.io, not this one
+    git push --tags
+    ```
+
+    Once this is pushed, go along to [the releases page on GitHub](https://github.com/paritytech/subxt/releases)
+    and draft a new release which points to the tag you just pushed to `master` above. Copy the changelog comments
+    for the current release into the release description.

cli/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "subxt-cli"
-version = "0.2.0"
+version = "0.16.0"
 authors = ["Parity Technologies <[email protected]>"]
 edition = "2021"
 
@@ -16,7 +16,7 @@ path = "src/main.rs"
 
 [dependencies]
 # perform subxt codegen
-subxt-codegen = { version = "0.2.0", path = "../codegen" }
+subxt-codegen = { version = "0.16.0", path = "../codegen" }
 # parse command line args
 structopt = "0.3.25"
 # make the request to a substrate node to get the metadata
@@ -36,4 +36,4 @@ scale = { package = "parity-scale-codec", version = "2.3.0", default-features =
 # handle urls to communicate with substrate nodes
 url = { version = "2.2.2", features = ["serde"] }
 # generate the item mod for codegen
-syn = "1.0.80"
+syn = "1.0.80"

cli/src/main.rs

@@ -1,4 +1,4 @@
-// Copyright 2019-2021 Parity Technologies (UK) Ltd.
+// Copyright 2019-2022 Parity Technologies (UK) Ltd.
 // This file is part of subxt.
 //
 // subxt is free software: you can redistribute it and/or modify
@@ -33,6 +33,7 @@ use std::{
     path::PathBuf,
 };
 use structopt::StructOpt;
+use subxt_codegen::GeneratedTypeDerives;
 
 /// Utilities for working with substrate metadata for subxt.
 #[derive(Debug, StructOpt)]
@@ -70,6 +71,9 @@ enum Command {
         /// the path to the encoded metadata file.
         #[structopt(short, long, parse(from_os_str))]
         file: Option<PathBuf>,
+        /// Additional derives
+        #[structopt(long = "derive")]
+        derives: Vec<String>,
     },
 }
 
@@ -102,7 +106,7 @@ fn main() -> color_eyre::Result<()> {
                 }
             }
         }
-        Command::Codegen { url, file } => {
+        Command::Codegen { url, file, derives } => {
             if let Some(file) = file.as_ref() {
                 if url.is_some() {
                     eyre::bail!("specify one of `--url` or `--file` but not both")
@@ -111,15 +115,15 @@ fn main() -> color_eyre::Result<()> {
                 let mut file = fs::File::open(file)?;
                 let mut bytes = Vec::new();
                 file.read_to_end(&mut bytes)?;
-                codegen(&mut &bytes[..])?;
+                codegen(&mut &bytes[..], derives)?;
                 return Ok(())
             }
 
             let url = url.unwrap_or_else(|| {
                 url::Url::parse("http://localhost:9933").expect("default url is valid")
             });
             let (_, bytes) = fetch_metadata(&url)?;
-            codegen(&mut &bytes[..])?;
+            codegen(&mut &bytes[..], derives)?;
             Ok(())
         }
     }
@@ -145,13 +149,24 @@ fn fetch_metadata(url: &url::Url) -> color_eyre::Result<(String, Vec<u8>)> {
     Ok((hex_data, bytes))
 }
 
-fn codegen<I: Input>(encoded: &mut I) -> color_eyre::Result<()> {
+fn codegen<I: Input>(
+    encoded: &mut I,
+    raw_derives: Vec<String>,
+) -> color_eyre::Result<()> {
     let metadata = <RuntimeMetadataPrefixed as Decode>::decode(encoded)?;
     let generator = subxt_codegen::RuntimeGenerator::new(metadata);
     let item_mod = syn::parse_quote!(
         pub mod api {}
     );
-    let runtime_api = generator.generate_runtime(item_mod, Default::default());
+
+    let p = raw_derives
+        .iter()
+        .map(|raw| syn::parse_str(raw))
+        .collect::<Result<Vec<_>, _>>()?;
+    let mut derives = GeneratedTypeDerives::default();
+    derives.append(p.into_iter());
+
+    let runtime_api = generator.generate_runtime(item_mod, derives);
     println!("{}", runtime_api);
     Ok(())
 }