WASM_BUILD_TYPE=debug fails to compile: UnknownOpcode(192)

[crystalin]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

Building the node-template with WASM_BUILD_TYPE=debug (to better support debugging it later) fails:

error: failed to run custom build command for `node-template-runtime v4.0.0-dev (projects/polkadot-sdk/substrate/bin/node-template/runtime)`

Caused by:
  process didn't exit successfully: `/home/crystalin/projects/polkadot-sdk/target/release/build/node-template-runtime-c9360a61cb5d0a14/build-script-build` (exit status: 1)
  --- stdout
  Information that should be included in a bug report.
  Executing build command: RUSTFLAGS="-C target-cpu=mvp -C target-feature=-sign-ext -C link-arg=--export-table -Clink-arg=--export=__heap_base -C link-arg=--import-memory  " SKIP_WASM_BUILD="" "/home/crystalin/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo" "rustc" "--target=wasm32-unknown-unknown" "--manifest-path=/home/crystalin/projects/polkadot-sdk/target/release/wbuild/node-template-runtime/Cargo.toml" "--color=always" "--profile" "dev"
  Using rustc version: rustc 1.72.1 (d5c2e9c34 2023-09-13)

  Failed to deserialize `/home/crystalin/projects/polkadot-sdk/target/release/wbuild/node-template-runtime/node_template_runtime.wasm`: UnknownOpcode(192)

  --- stderr
      Finished dev [unoptimized + debuginfo] target(s) in 0.11s

Steps to reproduce

WASM_BUILD_TYPE=debug cargo build --release -p node-template

[bkchr]

@koute it is back 😱

Should have been fixed by paritytech/substrate#13804 🤔

[koute]

Hm, this is weird, since it does show that -C target-feature=-sign-ext is there, so this opcode shouldn't be emitted, unless... the precompiled std that is shipped with Rust might have this opcode somewhere, and since this is a debug build LTO was not enabled so it wasn't stripped out.

I'll take a closer look.

Anyway, if that's the case then I can see two options:

1. We add support for the sign-ext extension. But this doesn't fundamentally solve the problem since the next time another extension gets pulled into the WASM core we'll see this again, e.g. I wouldn't be surprised if SIMD gets enabled by default in the future, and in that case we most likely wouldn't want to enable it.
2. Use -Z build-std and build std ourselves, which will increase compile times. (And technically it's unstable and nightly only, but we can use the RUSTC_BOOTSTRAP trick to work around this.)

[chrisdcosta]

Have you tried adding a rust-toolchain.toml file? It might help with the missing wasm32-unknown-unkown in your analysis.

Also check that you are using the latest version of substrate builder.

We had a similar issue, we were testing on an older version of Rococo local and initially could not upgrade with the Unknown Opcode(192) error.

We then started using the latest Rococo local and got upgrade rejections but no error.

#1824

[crystalin]

I tried with polkadot 1.1.0 which I think is using the latest substrate builder. I didn't try the polkadot-sdk build with the rust-toolchain.toml but when trying it on moonbeam it also fails to compile

[chrisdcosta]

Looking at the code it appears that the WASM_BUILD_TYPE defaults in most cases to Release builds regardless of the cargo build profile.

I have two questions then.

1. Why are you specifying a native build type of cargo build --release and the wasm build of debug?

2. What is the purpose of this misalignment?

[crystalin]

We use cargo build --release because it used to default to debug (long long time ago) and some parachain still default to debug.
Also debug generates a huge binary (> 2GB in Moonbeam IIRC) which makes it unusable in most cases.

Also the release can embed debug symbols which is enough for what I need. But it doesn't by default on the WASM so you need to use the WASM_BUILD_TYPE but it fails to build.

[chrisdcosta]

A quick update. This is where the error is triggered.

IIUC the binary should contain the version. This section is checking that the version exists. However, this line is where the deserialisation error occurs. You can check that by changing the process::exit(1) to something else.

So the file definitely exists otherwise it would not get that far. So something must happen to the "bloaty" wasm build.

Still looking further...

[koute]

So I finally looked into this.

As I expected, these opcodes come from the precompiled standard library supplied with Rust, more specifically:

(func core::str::converts::from_utf8::hfb9df7006e72565a (;2551;) (type 3) (param i32 i32 i32)
...
i32.extend8_s

and

(func core::unicode::printable::check::h18fc1c66e5a57e60 (;2604;) (type 54) (param i32 i32 i32 i32 i32 i32 i32) (result i32)
...
i32.extend8_s

where the i32.extend8_s is the instruction with the 192 opcode and is part of the sign-ext WASM extension which was pulled into the WASM core (so it isn't an extension anymore).

It only accidentally works when the runtime is built in release mode because those functions do not use this instruction then, most likely due to one of the LLVM optimization passes eliminating them altogether.

I think the most reasonable course of action here is probably to just compile the runtimes with -Z build-std. I'll try to whip up a PR with a fix.

---

Of course another alternative would be to support the sign-ext extension (and whatever else was also pulled in into the newest WASM spec), but that would be a lot more involved. To do this properly and in a safe fashion we'd have to:

1. Write a Polkadot fellows RFC (since the requirement to support this affects the core protocol).
2. Add a mechanism to atomically enable support for this extension in a coordinated fashion.
   • Release a new node version with a configuration knob that can enable this extension.
   • Expose this configuration knob on chain.
   • Wait until the vast majority of nodes are upgraded.
   • Flip the switch on chain to atomically enable it.

Personally I'm not convinced it's worth it though. That's a lot of work and extra complexity for something that'll end up most likely being practically useless. And it's probably just a matter of time until an extension which we actively do not want gets pulled into the WASM core and made the default.