Skip to content
This repository was archived by the owner on Nov 15, 2023. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
886 changes: 485 additions & 401 deletions Cargo.lock

Large diffs are not rendered by default.

24 changes: 12 additions & 12 deletions bin/node-template/node/src/command.rs
Original file line number Diff line number Diff line change
Expand Up @@ -47,36 +47,36 @@ impl SubstrateCli for Cli {
2017
}

fn load_spec(&self, id: &str) -> Result<Box<dyn sc_service::ChainSpec>, String> {
Ok(match id {
"dev" => Box::new(chain_spec::development_config()?),
"" | "local" => Box::new(chain_spec::local_testnet_config()?),
path => Box::new(chain_spec::ChainSpec::from_json_file(
std::path::PathBuf::from(path),
)?),
})
}

fn native_runtime_version(_: &Box<dyn ChainSpec>) -> &'static RuntimeVersion {
&node_template_runtime::VERSION
}
}

fn spec_factory(id: String) -> Result<Box<dyn ChainSpec>, String> {
Ok(match &*id {
"dev" => Box::new(chain_spec::development_config()?),
"" | "local" => Box::new(chain_spec::local_testnet_config()?),
path => Box::new(chain_spec::ChainSpec::from_json_file(
std::path::PathBuf::from(path),
)?),
})
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are you moving that out again? It wasn't nice inside SubstrateCli?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It wasn't nice inside SubstrateCli?

unfortunately, having the spec_factory in the cli complicates the API for paritytech/substrate-test-runner, where users would also have to provide impl SubstrateCli simply because we want the spec_factory. Splitting it out keeps our API lean and simple.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok ok

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it related in any way with this? #6651

Are you trying to run nodes inside tests?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@seunlanlege I don't see the full picture yet so bare with me I might be completely wrong. If you can provide me documentation and more information on the project it would help me understand better the big picture.

Wild guess: I don't think you need sc-cli at all. sc-cli is for building clis, here you are building a library, you should be using sc-service and use the sc_service::Configuration object directly. It's sc-service that is the glue code and the center of everything, not sc-cli. You can't just pick the elements you want from sc-cli without applying to the full logic of sc-cli's usage.

Example: if you pass the arguments of the cli by code, I'm confident that this is wrong. sc-cli is really only meant for building a binary and the arguments are passed by a user.

impl<Runtime> InternalNode<Runtime> {
	pub fn builder() -> InternalNodeBuilder<Runtime> {
		InternalNodeBuilder::new()
	}

	pub fn new(logs: Logger, cli: &[String]) -> Self {
		let cli = node_cli::Cli::from_iter(cli.iter());
...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(In other words: sc-cli must be a dependency of a bin crate, not a lib crate)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cecton Please show them your code you have done for the polkadot test service. Which solves this problem AFAIK.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/paritytech/polkadot/blob/master/node/test-service/src/lib.rs

@seunlanlege This is a testing crate for polkadot. It uses sc-service directly. I just pass the chain-spec I want, tweak the sc_service::Configuration object the way I want, and that's how I can run a node. The nodes can communicate with each other using memory network so it doesn't open a port on the machine and I don't need to know the port number.

The PR #6651 is meant to improve testing nodes by providing an async runtime for the test and a timeout.

The PR #6555 is also meant to improve testing nodes by providing a "wait_for_blocks" function (to check that blocks are being created) and "send_transaction" to arbitrarily send a transaction to the node.


/// Parse and run command line arguments
pub fn run() -> sc_cli::Result<()> {
let cli = Cli::from_args();

match &cli.subcommand {
Some(subcommand) => {
let runner = cli.create_runner(subcommand)?;
let runner = cli.create_runner(subcommand, spec_factory)?;
runner.run_subcommand(subcommand, |config| {
let PartialComponents { client, backend, task_manager, import_queue, .. }
= new_partial(&config)?;
Ok((client, backend, import_queue, task_manager))
})
}
None => {
let runner = cli.create_runner(&cli.run)?;
let runner = cli.create_runner(&cli.run, spec_factory)?;
runner.run_node_until_exit(|config| match config.role {
Role::Light => service::new_light(config),
_ => service::new_full(config),
Expand Down
2 changes: 1 addition & 1 deletion bin/node-template/node/src/service.rs
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ pub fn new_full(config: Configuration) -> Result<TaskManager, ServiceError> {
} = new_partial(&config)?;

let finality_proof_provider =
GrandpaFinalityProofProvider::new_for_service(backend.clone(), client.clone());
GrandpaFinalityProofProvider::new_for_service(backend.clone(),client.clone());

let (network, network_status_sinks, system_rpc_tx, network_starter) =
sc_service::build_network(sc_service::BuildNetworkParams {
Expand Down
32 changes: 16 additions & 16 deletions bin/node/cli/src/command.rs
Original file line number Diff line number Diff line change
Expand Up @@ -48,43 +48,43 @@ impl SubstrateCli for Cli {
2017
}

fn load_spec(&self, id: &str) -> std::result::Result<Box<dyn sc_service::ChainSpec>, String> {
Ok(match id {
"dev" => Box::new(chain_spec::development_config()),
"local" => Box::new(chain_spec::local_testnet_config()),
"" | "fir" | "flaming-fir" => Box::new(chain_spec::flaming_fir_config()?),
"staging" => Box::new(chain_spec::staging_testnet_config()),
path => Box::new(chain_spec::ChainSpec::from_json_file(
std::path::PathBuf::from(path),
)?),
})
}

fn native_runtime_version(_: &Box<dyn ChainSpec>) -> &'static RuntimeVersion {
&node_runtime::VERSION
}
}

fn spec_factory(id: String) -> std::result::Result<Box<dyn sc_service::ChainSpec>, String> {
Ok(match &*id {
"dev" => Box::new(chain_spec::development_config()),
"local" => Box::new(chain_spec::local_testnet_config()),
"" | "fir" | "flaming-fir" => Box::new(chain_spec::flaming_fir_config()?),
"staging" => Box::new(chain_spec::staging_testnet_config()),
path => Box::new(chain_spec::ChainSpec::from_json_file(
std::path::PathBuf::from(path),
)?),
})
}

/// Parse command line arguments into service configuration.
pub fn run() -> Result<()> {
let cli = Cli::from_args();

match &cli.subcommand {
None => {
let runner = cli.create_runner(&cli.run)?;
let runner = cli.create_runner(&cli.run, spec_factory)?;
runner.run_node_until_exit(|config| match config.role {
Role::Light => service::new_light(config),
_ => service::new_full(config),
})
}
Some(Subcommand::Inspect(cmd)) => {
let runner = cli.create_runner(cmd)?;
let runner = cli.create_runner(cmd, spec_factory)?;

runner.sync_run(|config| cmd.run::<Block, RuntimeApi, Executor>(config))
}
Some(Subcommand::Benchmark(cmd)) => {
if cfg!(feature = "runtime-benchmarks") {
let runner = cli.create_runner(cmd)?;
let runner = cli.create_runner(cmd, spec_factory)?;

runner.sync_run(|config| cmd.run::<Block, Executor>(config))
} else {
Expand All @@ -94,7 +94,7 @@ pub fn run() -> Result<()> {
}
}
Some(Subcommand::Base(subcommand)) => {
let runner = cli.create_runner(subcommand)?;
let runner = cli.create_runner(subcommand, spec_factory)?;
runner.run_subcommand(subcommand, |config| {
let PartialComponents { client, backend, task_manager, import_queue, ..}
= new_partial(&config)?;
Expand Down
2 changes: 1 addition & 1 deletion bin/node/cli/src/service.rs
Original file line number Diff line number Diff line change
Expand Up @@ -335,7 +335,7 @@ pub fn new_full(config: Configuration)
}

pub fn new_light_base(config: Configuration) -> Result<(
TaskManager, Arc<RpcHandlers>, Arc<LightClient>,
TaskManager, RpcHandlers, Arc<LightClient>,
Arc<NetworkService<Block, <Block as BlockT>::Hash>>,
Arc<sc_transaction_pool::LightPool<Block, LightClient, sc_network::config::OnDemand<Block>>>
), ServiceError> {
Expand Down
28 changes: 24 additions & 4 deletions client/cli/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -403,14 +403,18 @@ pub trait CliConfiguration: Sized {
}

/// Create a Configuration object from the current object
fn create_configuration<C: SubstrateCli>(
fn create_configuration<C, F>(
&self,
cli: &C,
factory: F,
task_executor: TaskExecutor,
) -> Result<Configuration> {
) -> Result<Configuration>
where
C: SubstrateCli,
F: ChainSpecFactory,
{
let is_dev = self.is_dev()?;
let chain_id = self.chain_id(is_dev)?;
let chain_spec = cli.load_spec(chain_id.as_str())?;
let chain_spec = factory.load_spec(chain_id)?;
let base_path = self
.base_path()?
.unwrap_or_else(|| BasePath::from_project("", "", &C::executable_name()));
Expand Down Expand Up @@ -507,6 +511,22 @@ pub trait CliConfiguration: Sized {
}
}

/// Chain spec factory, allows for creating different chain specs, given different
/// chain_id's.
pub trait ChainSpecFactory {
/// produces a chain spec given a valid chain_id.
fn load_spec(&self, id: String) -> std::result::Result<Box<dyn sc_service::ChainSpec>, String>;
}

impl<F> ChainSpecFactory for F
where
F: Fn(String) -> std::result::Result<Box<dyn sc_service::ChainSpec>, String>
{
fn load_spec(&self, id: String) -> std::result::Result<Box<dyn sc_service::ChainSpec>, String> {
(self)(id)
}
}

/// Generate a valid random name for the node
pub fn generate_node_name() -> String {
loop {
Expand Down
22 changes: 14 additions & 8 deletions client/cli/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -88,9 +88,6 @@ pub trait SubstrateCli: Sized {
/// Copyright starting year (x-current year)
fn copyright_start_year() -> i32;

/// Chain spec factory
fn load_spec(&self, id: &str) -> std::result::Result<Box<dyn ChainSpec>, String>;

/// Helper function used to parse the command line arguments. This is the equivalent of
/// `structopt`'s `from_iter()` except that it takes a `VersionInfo` argument to provide the name of
/// the application, author, "about" and version. It will also set `AppSettings::GlobalVersion`.
Expand Down Expand Up @@ -209,19 +206,28 @@ pub trait SubstrateCli: Sized {
}

/// Only create a Configuration for the command provided in argument
fn create_configuration<T: CliConfiguration>(
fn create_configuration<T, F>(
&self,
command: &T,
factory: F,
task_executor: TaskExecutor,
) -> error::Result<Configuration> {
command.create_configuration(self, task_executor)
) -> Result<Configuration>
where
T: CliConfiguration,
F: ChainSpecFactory,
{
command.create_configuration::<Self, F>(factory, task_executor)
}

/// Create a runner for the command provided in argument. This will create a Configuration and
/// a tokio runtime
fn create_runner<T: CliConfiguration>(&self, command: &T) -> error::Result<Runner<Self>> {
fn create_runner<C, F>(&self, command: &C, factory: F) -> error::Result<Runner<Self>>
where
C: CliConfiguration,
F: ChainSpecFactory,
{
command.init::<Self>()?;
Runner::new(self, command)
Runner::new::<C, F>(command, factory)
}

/// Native runtime version.
Expand Down
10 changes: 7 additions & 3 deletions client/cli/src/runner.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
// You should have received a copy of the GNU General Public License
// along with this program. If not, see <https://www.gnu.org/licenses/>.

use crate::CliConfiguration;
use crate::{CliConfiguration, ChainSpecFactory};
use crate::Result;
use crate::Subcommand;
use crate::SubstrateCli;
Expand Down Expand Up @@ -122,7 +122,11 @@ pub struct Runner<C: SubstrateCli> {

impl<C: SubstrateCli> Runner<C> {
/// Create a new runtime with the command provided in argument
pub fn new<T: CliConfiguration>(cli: &C, command: &T) -> Result<Runner<C>> {
pub fn new<T, F>(command: &T, factory: F) -> Result<Runner<C>>
where
T: CliConfiguration,
F: ChainSpecFactory,
{
let tokio_runtime = build_runtime()?;
let runtime_handle = tokio_runtime.handle().clone();

Expand All @@ -136,7 +140,7 @@ impl<C: SubstrateCli> Runner<C> {
};

Ok(Runner {
config: command.create_configuration(cli, task_executor.into())?,
config: command.create_configuration::<C, F>(factory, task_executor.into())?,
tokio_runtime,
phantom: PhantomData,
})
Expand Down
5 changes: 3 additions & 2 deletions client/consensus/manual-seal/src/rpc.rs
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ impl<Hash> ManualSeal<Hash> {
}
}

impl<Hash: Send + 'static> ManualSealApi<Hash> for ManualSeal<Hash> {
impl<Hash: std::fmt::Debug + Send + 'static> ManualSealApi<Hash> for ManualSeal<Hash> {
fn create_block(
&self,
create_empty: bool,
Expand All @@ -122,7 +122,8 @@ impl<Hash: Send + 'static> ManualSealApi<Hash> for ManualSeal<Hash> {
sender: Some(sender),
};
sink.send(command).await?;
receiver.await?
let res = receiver.await?;
res
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are these changes about?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stray debug changes

}.boxed();

Box::new(future.map_err(Error::from).compat())
Expand Down
3 changes: 2 additions & 1 deletion client/service/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,8 @@ test-helpers = []
derive_more = "0.99.2"
futures01 = { package = "futures", version = "0.1.29" }
futures = { version = "0.3.4", features = ["compat"] }
jsonrpc-pubsub = "14.2.0"
jsonrpc-pubsub = "14.2"
jsonrpc-core = "14.2"
rand = "0.7.3"
parking_lot = "0.10.0"
lazy_static = "1.4.0"
Expand Down
4 changes: 2 additions & 2 deletions client/service/src/builder.rs
Original file line number Diff line number Diff line change
Expand Up @@ -442,7 +442,7 @@ pub fn build_offchain_workers<TBl, TBackend, TCl>(
/// Spawn the tasks that are required to run a node.
pub fn spawn_tasks<TBl, TBackend, TExPool, TRpc, TCl>(
params: SpawnTasksParams<TBl, TCl, TExPool, TRpc, TBackend>,
) -> Result<Arc<RpcHandlers>, Error>
) -> Result<RpcHandlers, Error>
where
TCl: ProvideRuntimeApi<TBl> + HeaderMetadata<TBl, Error=sp_blockchain::Error> + Chain<TBl> +
BlockBackend<TBl> + BlockIdTo<TBl, Error=sp_blockchain::Error> + ProofProvider<TBl> +
Expand Down Expand Up @@ -540,7 +540,7 @@ pub fn spawn_tasks<TBl, TBackend, TExPool, TRpc, TCl>(
);
let rpc = start_rpc_servers(&config, gen_handler)?;
// This is used internally, so don't restrict access to unsafe RPC
let rpc_handlers = Arc::new(RpcHandlers(gen_handler(sc_rpc::DenyUnsafe::No)));
let rpc_handlers = RpcHandlers(Arc::new(gen_handler(sc_rpc::DenyUnsafe::No).into()));

// Telemetry
let telemetry = config.telemetry_endpoints.clone().and_then(|endpoints| {
Expand Down
9 changes: 8 additions & 1 deletion client/service/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,8 @@ impl<T: MallocSizeOf> MallocSizeOfWasm for T {}
impl<T> MallocSizeOfWasm for T {}

/// RPC handlers that can perform RPC queries.
pub struct RpcHandlers(sc_rpc_server::RpcHandler<sc_rpc::Metadata>);
#[derive(Clone)]
pub struct RpcHandlers(Arc<jsonrpc_core::MetaIoHandler<sc_rpc::Metadata>>);

impl RpcHandlers {
/// Starts an RPC query.
Expand All @@ -115,8 +116,14 @@ impl RpcHandlers {
.map(|res| res.expect("this should never fail"))
.boxed()
}

/// Provides access to the underlying `MetaIoHandler`
pub fn io_handler(&self) -> Arc<jsonrpc_core::MetaIoHandler<sc_rpc::Metadata>> {
self.0.clone()
}
Comment on lines +121 to +123
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you absolutely need this function, or can you add the MetaIoHandler methods directly to RpcHandlers? If so, that would be cleaner and more abstract IMO.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok so the use case for this is to allow end users (and me 😇 ) use the jsonrpc-core-client local transport, which takes a Deref<Target=MetaIoHandler> + 'static. Hence the need to have the MetaIoHandler behind an Arc.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, fair enough : )

}


/// Sinks to propagate network status updates.
/// For each element, every time the `Interval` fires we push an element on the sender.
#[derive(Clone)]
Expand Down
9 changes: 6 additions & 3 deletions client/service/src/task_manager/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -336,9 +336,12 @@ impl TaskManager {
}
}

/// Set what the task manager should keep alive.
pub(super) fn keep_alive<T: 'static + Send + Sync>(&mut self, to_keep_alive: T) {
self.keep_alive = Box::new(to_keep_alive);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@expenses I was sure that you did make a tuple magic thingy here that would make this function callable multiple times 🤔 why was is it gone? I'm missing something...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean keep_alive((SOmehing, else, yolo))

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I totally mean S0mething, else, yolo.

I think she refactored her code before pushing and since keep_alive here is privat-ish she didn't foresee it could be misused.

Clearly something to fix, it's a pitfall.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah SOmething :P You should change your font to distinguish O and 0 :P

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The mix between the monospace font and sans font tricked my eyes

/// Set what the task manager should keep alive, can be called multiple times.
pub fn keep_alive<T: 'static + Send + Sync>(&mut self, to_keep_alive: T) {
// allows this fn to safely called multiple times.
use std::mem;
let old = mem::replace(&mut self.keep_alive, Box::new(()));
self.keep_alive = Box::new((to_keep_alive, old));
Comment on lines -339 to +344
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you absolutely need to call this function more than once, then this is fine..

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's just to tempting to call it more than once imo. If there is no downside we should do it

}

/// Register another TaskManager to terminate and gracefully shutdown when the parent
Expand Down
4 changes: 2 additions & 2 deletions utils/browser/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ use wasm_bindgen::prelude::*;
use futures::{
prelude::*, channel::{oneshot, mpsc}, compat::*, future::{ready, ok, select}
};
use std::{sync::Arc, pin::Pin};
use std::pin::Pin;
use sc_chain_spec::Extension;
use libp2p_wasm_ext::{ExtTransport, ffi};

Expand Down Expand Up @@ -124,7 +124,7 @@ struct RpcMessage {
}

/// Create a Client object that connects to a service.
pub fn start_client(mut task_manager: TaskManager, rpc_handlers: Arc<RpcHandlers>) -> Client {
pub fn start_client(mut task_manager: TaskManager, rpc_handlers: RpcHandlers) -> Client {
// We dispatch a background task responsible for processing the service.
//
// The main action performed by the code below consists in polling the service with
Expand Down