Skip to content
This repository was archived by the owner on Nov 15, 2023. It is now read-only.

Conversation

@athei
Copy link
Member

@athei athei commented Aug 20, 2021

In addition to the extensive synthetic benchmark suite we have in contracts that is used to determine weights we want to add a set of benchmarks that use real world benchmarks. That will give us a tool to compare the progress of different contract languages (ink, solang) and execution engines (wasmi, wasmer).

This PR creates some infrastructure code to allow that benchmarks to be written and adds the first two benchmarks:

  • erc20 transfer using the ink! erc20 example contract
  • erc20 transfer using the openzeppelin erc20 contract compiled with solang

You can run those benchmarks by running this commandline in the substrate repo:

cargo run --release --features runtime-benchmarks,[wasmer-sandbox] -- --extra --dev --execution=wasm --wasm-execution=compiled -p pallet_contracts -e ink_erc20_transfer

The wasmer-sandbox feature is optional and can be used to run the benchmark using wasmer. I ran both benchmarks using wasmi and wasmer. The g parameter describes whether gas instrumentation is injected or not. These are the results from running it on a rather slow x64 linux:

ink_erc20_transfer, wasmi

Min Squares Analysis
========
-- Extrinsic Time --

Data points distribution:
    g   mean µs  sigma µs       %
    0      5949     21.06    0.3%
    1      8795     53.03    0.6%

Quality and confidence:
param     error
g         8.151

Model:
Time ~=     5949
    + g     2845
              µs


ink_erc20_transfer, wasmer

Min Squares Analysis
========
-- Extrinsic Time --

Data points distribution:
    g   mean µs  sigma µs       %
    0      6010     11.97    0.1%
    1      9453     15.15    0.1%

Quality and confidence:
param     error
g         2.759

Model:
Time ~=     6010
    + g     3442
              µs



solang_erc20_transfer, wasmi

Min Squares Analysis
========
-- Extrinsic Time --

Data points distribution:
    g   mean µs  sigma µs       %
    0      2050     9.492    0.4%
    1      2179     13.77    0.6%

Quality and confidence:
param     error
g         2.389

Model:
Time ~=     2050
    + g    128.9
              µs


solang_erc20_transfer, wasmer

Min Squares Analysis
========
-- Extrinsic Time --

Data points distribution:
    g   mean µs  sigma µs       %
    0      2046     9.061    0.4%
    1      2134     5.513    0.2%

Quality and confidence:
param     error
g         1.515

Model:
Time ~=     2046
    + g    88.56
              µs

@athei athei added A0-please_review Pull request needs code review. B0-silent Changes should not be mentioned in any release notes C1-low PR touches the given topic and has a low impact on builders. D2-notlive 💤 PR contains changes in a runtime directory that is not deployed to a chain that requires an audit. labels Aug 20, 2021
@athei athei requested a review from cmichi August 20, 2021 12:48
Copy link
Contributor

@cmichi cmichi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Co-authored-by: Michael Müller <[email protected]>
@athei athei requested review from ascjones and gui1117 August 30, 2021 14:48
@athei
Copy link
Member Author

athei commented Sep 6, 2021

bot merge

@ghost
Copy link

ghost commented Sep 6, 2021

Trying merge.

@ghost ghost merged commit 47ee91e into master Sep 6, 2021
@ghost ghost deleted the at-macro-bench branch September 6, 2021 11:30
This pull request was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

A0-please_review Pull request needs code review. B0-silent Changes should not be mentioned in any release notes C1-low PR touches the given topic and has a low impact on builders. D2-notlive 💤 PR contains changes in a runtime directory that is not deployed to a chain that requires an audit.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants