thread 'tokio-runtime-worker-1' panicked at 'Bad input data provided to execute_block', node/runtime/src/lib.rs:421:1

[algys]

Thread panic error after upgrade runtime code via 'upgradeKey' method.

Reproduce steps:

• run node cargo run -- --dev --chain local
• change interface of runtime module, e.g. add second param for 'set' method in 'timestamp' module
• build wasm runtime and send 'upgradeKey'

logs:

2018-12-05 01:52:20 tokio-runtime-worker-1 WARN substrate_client::client     Function "inherent_extrinsics"
2018-12-05 01:52:20 tokio-runtime-worker-1 WARN substrate_client::client     Native result Ok([8, 32, 1, 0, 0, 3, 35, 5, 7, 92, 16, 1, 1, 1, 0])
2018-12-05 01:52:20 tokio-runtime-worker-1 WARN substrate_client::client     Wasm result Ok([8, 64, 1, 0, 0, 3, 35, 5, 7, 92, 0, 0, 0, 0, 0, 0, 0, 0, 16, 1, 1, 1, 0])
2018-12-05 01:52:21 tokio-runtime-worker-0 DEBUG sync  Propagating extrinsics
2018-12-05 01:52:21 tokio-runtime-worker-2 INFO substrate  Idle (0 peers), best: #227 (0x667c…f1bc)
2018-12-05 01:52:22 tokio-runtime-worker-1 DEBUG txpool  Banning invalid transactions: []
2018-12-05 01:52:22 tokio-runtime-worker-1 INFO substrate_service::consensus  Proposing block [number: 228; hash: 0x5348…9e07; parent_hash: 0x667c…f1bc; extrinsics: [0x2819…2c75, 0x7b4e…2d59]]
thread 'tokio-runtime-worker-1' panicked at 'Bad input data provided to execute_block', node/runtime/src/lib.rs:421:1
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::print
             at libstd/sys_common/backtrace.rs:71
             at libstd/sys_common/backtrace.rs:59
   2: std::panicking::default_hook::{{closure}}
             at libstd/panicking.rs:211
   3: std::panicking::default_hook
             at libstd/panicking.rs:227
   4: std::panicking::rust_panic_with_hook
             at libstd/panicking.rs:477
   5: std::panicking::continue_panic_fmt
             at libstd/panicking.rs:391
   6: std::panicking::begin_panic_fmt
             at libstd/panicking.rs:346
   7: node_runtime::api::dispatch
             at ./<::client::runtime_api::macros::impl_runtime_apis macros>:72
   8: <node_executor::Executor as substrate_executor::native_executor::NativeExecutionDispatch>::dispatch::{{closure}}
             at ./<::substrate_executor::native_executor::native_executor_instance macros>:15
   9: std::panicking::try::do_call
             at libstd/panicking.rs:310
  10: __rust_maybe_catch_panic
             at libpanic_unwind/lib.rs:103
  11: std::panicking::try
             at libstd/panicking.rs:289
  12: std::panic::catch_unwind
             at libstd/panic.rs:392
  13: substrate_executor::native_executor::safe_call
             at ./core/executor/src/native_executor.rs:85
  14: substrate_executor::native_executor::with_native_environment::{{closure}}
             at ./core/executor/src/native_executor.rs:96
  15: environmental::using::{{closure}}
             at /home/algys/.cargo/registry/src/github.amrom.workers.dev-1ecc6299db9ec823/environmental-1.0.0/src/lib.rs:90
  16: <std::thread::local::LocalKey<T>>::try_with
             at libstd/thread/local.rs:294
  17: <std::thread::local::LocalKey<T>>::with
             at libstd/thread/local.rs:248
  18: environmental::using
             at /home/algys/.cargo/registry/src/github.amrom.workers.dev-1ecc6299db9ec823/environmental-1.0.0/src/lib.rs:67
  19: sr_io::ext::using
             at ./<::environmental::environmental macros>:21
  20: sr_io::with_externalities
             at ./core/sr-io/src/../with_std.rs:207
  21: substrate_executor::native_executor::with_native_environment
             at ./core/executor/src/native_executor.rs:96
  22: <node_executor::Executor as substrate_executor::native_executor::NativeExecutionDispatch>::dispatch
             at ./<::substrate_executor::native_executor::native_executor_instance macros>:14
  23: <substrate_executor::native_executor::NativeExecutor<D> as substrate_state_machine::CodeExecutor<substrate_primitives::hasher::blake2::Blake2Hasher>>::call
             at ./core/executor/src/native_executor.rs:191
  24: substrate_state_machine::execute_using_consensus_failure_handler
             at ./core/state-machine/src/lib.rs:322
  25: <substrate_client::call_executor::LocalCallExecutor<B, E> as substrate_client::call_executor::CallExecutor<Block, substrate_primitives::hasher::blake2::Blake2Hasher>>::call_at_state
             at ./core/client/src/call_executor.rs:170
  26: <substrate_client::client::Client<B, E, Block, RA>>::execute_and_import_block
             at ./core/client/src/client.rs:635
  27: <substrate_client::client::Client<B, E, Block, RA> as substrate_consensus_common::block_import::BlockImport<Block>>::import_block
             at ./core/client/src/client.rs:1106
  28: <substrate_finality_grandpa::GrandpaBlockImport<B, E, Block, RA, PRA> as substrate_consensus_common::block_import::BlockImport<Block>>::import_block
             at ./core/finality-grandpa/src/lib.rs:666
  29: substrate_consensus_aura::start_aura::{{closure}}::{{closure}}::{{closure}}
             at ./core/consensus/aura/src/lib.rs:253
  30: <core::result::Result<T, E>>::map
   ....

full bt: link

[gnunicorn]

Hey @algys

Reproduce steps:

• run node cargo run -- --dev --chain local
• change interface of runtime module, e.g. add second param for 'set' method in 'timestamp' module
• build wasm runtime and send 'upgradeKey'

Sounds like you didn't change the runtime version before compiling/uploading the new runtime. From the logs it appears that because of that the runtime update is not considered different and the native version is being executed. However, as the native version doesn't have that new parameter, it fails to execute the newly changed transactions. Please update the runtime version and try again on a fresh chain. If the error persists please feel free to reopen this ticket!

[algys]

You right! I didn't change the version before upgrade.

But now I tried change version before upgrade on a fresh chain:

When I change impl_version in runtime before update I got differ error:

2018-12-14 19:47:25 Starting consensus session on top of parent 0x5ffa7b985061fc632ac7f1ab348fe598c1743b193b35e9867ebaa9c737b47dbb
2018-12-14 19:47:27 Consensus error between wasm and native runtime execution at block Hash(0x5ffa7b985061fc632ac7f1ab348fe598c1743b193b35e9867ebaa9c737b47dbb)
2018-12-14 19:47:27    Function "inherent_extrinsics"
2018-12-14 19:47:27    Native result Ok([8, 32, 1, 0, 0, 3, 157, 222, 19, 92, 16, 1, 1, 1, 0])
2018-12-14 19:47:27    Wasm result Ok([8, 64, 1, 0, 0, 3, 157, 222, 19, 92, 0, 0, 0, 0, 0, 0, 0, 0, 16, 1, 1, 1, 0])
/home/algys/sources/substrate/srml/timestamp/src/lib.rs
84
4
2018-12-14 19:47:28 Consensus error between wasm and native runtime execution at block Hash(0x5ffa7b985061fc632ac7f1ab348fe598c1743b193b35e9867ebaa9c737b47dbb)

Then I did change all runtime versions(on a fresh chain):

pub const VERSION: RuntimeVersion = RuntimeVersion {
	spec_name: ver_str!("node"),
	impl_name: ver_str!("substrate-node"),
	authoring_version: 2,
	spec_version: 2,
	impl_version: 2,
	apis: apis_vec!([
		(BLOCK_BUILDER, 2),
		(TAGGED_TRANSACTION_QUEUE, 2),
		(METADATA, 2),
		(GRANDPA_API, 2),
	]),
};

and got another different error:

2018-12-14 20:01:36 Starting consensus session on top of parent 0xc5e3d86b67183bbc0fd3c5cfe9cb658ce9b057e1a37b54211d829b47359283d1
2018-12-14 20:01:36 Idle (0 peers), best: #14 (0xc5e3…83d1)
/home/algys/sources/substrate/srml/timestamp/src/lib.rs
101
4
2018-12-14 20:01:37 Failed to construct block: Error(Execution(Error(Wasmi(Trap(Trap { kind: Unreachable })), State { next_error: None, backtrace: InternalBacktrace { backtrace: None } })), State { next_error: None, backtrace: InternalBacktrace { backtrace: None } })
2018-12-14 20:01:37 Aura authorship task terminated unexpectedly. Restarting
2018-12-14 20:01:41 Idle (0 peers), best: #14 (0xc5e3…83d1)

Before both tests I wiped chain data and start on a fresh chain

[gnunicorn]

impl_version isn't enough if you change the API, as it implies that this is "minor" change, that either is about a bug or performance issues and thus, the native version can still be run. Only if spec_version is different the current check will for sure use the wasm version rather than native. Lastly, the authoring_version should not actually change because authoring didn't change and is required to match with the outer version otherwise, a proposer can't be sure to still be authoring in a way the runtime is able to verify.

Thus, all you want to change is spec_version and then do an update :) .

[algys]

Ok, thank you, finally I understood, but problem still actual

I incremented only spec_version and got same error:

2018-12-14 20:39:20 Idle (0 peers), best: #10 (0x78ab…0ab9)
2018-12-14 20:39:24 Starting consensus session on top of parent 0x78abe5f6a533d4f9a90cd4064469b1b28be6e50ba81bc22e30a394027da20ab9
/home/algys/sources/substrate/srml/timestamp/src/lib.rs
84
4
2018-12-14 20:39:25 Failed to construct block: Error(Execution(Error(Wasmi(Trap(Trap { kind: Unreachable })), State { next_error: None, backtrace: InternalBacktrace { backtrace: None } })), State { next_error: None, backtrace: InternalBacktrace { backtrace: None } })