Multiple Enhancements

.github/workflows/test.yaml

@@ -9,7 +9,7 @@ jobs:
       matrix:
         python: ['3.10']
         golang: [1.17]
-
+        solc: ['0.8.17']
     steps:
       - uses: actions/checkout@v2
         with:
@@ -33,6 +33,12 @@ jobs:
         uses: actions/setup-python@v2
         with:
           python-version: ${{ matrix.python }}
+      - name: Install solc compiler
+        run: |
+          RELEASE_NAME=$(curl https://binaries.soliditylang.org/linux-amd64/list.json | jq -r --arg SOLC_VERSION "${{ matrix.solc }}" '.releases[$SOLC_VERSION]')
+          wget -O $GITHUB_WORKSPACE/bin/solc https://binaries.soliditylang.org/linux-amd64/$RELEASE_NAME
+          chmod a+x $GITHUB_WORKSPACE/bin/solc
+          echo $GITHUB_WORKSPACE/bin >> $GITHUB_PATH
       - name: Install Tox and any other packages
         run: pip install tox requests
       - name: Run Tox (CPython)

.gitignore

@@ -7,6 +7,8 @@ __pycache__
 __pycache__/
 *.py[cod]
 *$py.class
+venv/
+fixtures/
 
 # C extensions
 *.so
@@ -44,3 +46,7 @@ pip-delete-this-directory.txt
 /doc/_autosummary
 
 .coverage
+
+# misc
+
+*.code-workspace

README.md

@@ -5,7 +5,7 @@ Ethereum tests.
 
 ## Quick Start
 
-Relies on Python `3.10.0` and `geth` `v1.10.13` or later. 
+Relies on Python `3.10.0`, `geth` `v1.10.13`, `solc` `v0.8.5` or later. 
 
 ```console
 $ git clone https://github.com/lightclient/testing-tools
@@ -36,5 +36,142 @@ This is a wrapper around the [block builder][b11r] (b11r) tool.
 
 This is a wrapper around the [transaction][t8n] (t8n) tool.
 
+### `ethereum_tests`
+
+Contains all the Ethereum consensus tests available in this repository.
+
+## Writing Tests
+
+### Adding a New Test
+
+All currently implemented tests can be found in the `src/ethereum_tests`
+directory, which is composed of many subdirectories, and each one represents a
+different test category.
+
+Source files included in each category contain one or multiple test functions,
+and each can in turn create one or many test vectors.
+
+A new test can be added by either:
+
+- Adding a new `test_` function to an existing file in any of the existing
+  category subdirectories within `src/ethereum_tests`.
+- Creating a new source file in an existing category, and populating it with
+  the new test function(s).
+- Creating an entirely new category by adding a subdirectory in
+  `src/ethereum_tests` with the appropriate source files and test functions.
+
+### Test Generator Functions
+
+Every test function is a generator which can perform a single or multiple
+`yield` operations during its runtime to each time yield a single `StateTest` 
+object.
+
+The test vector's generator function _must_ be decorated by only one of the
+following decorators:
+- test_from
+- test_from_until
+- test_only
+
+These decorators specify the forks on which the test vector is supposed to run.
+
+They also automatically append necessary information for the
+`ethereum_test_filler` to process when the generator is being executed to fill
+the tests.
+
+The test vector function must take only one `str` parameter: the fork name.
+
+### `StateTest` Object
+
+The `StateTest` object represents a single test vector, and contains the
+following attributes:
+
+- env: Environment object which describes the global state of the blockchain
+    before the test starts.
+- pre: Pre-State containing the information of all Ethereum accounts that exist
+    before any transaction is executed.
+- post: Post-State containing the information of all Ethereum accounts that are
+    created or modified after all transactions are executed.
+- txs: All transactions to be executed during the test vector runtime.
+
+
+### Pre/Post State of the Test
+
+The `pre` and `post` states are elemental to setup and then verify the outcome
+of the state test.
+
+Both `pre` and `post` are mappings of account addresses to `account` structures:
+```
+class Account:
+    nonce: int
+    balance: int
+    code: Union[bytes, str, Code]
+    storage: Storage
+```
+
+A single test vector can contain as many accounts in the `pre` and `post` states
+as required, and they can be also filled dynamically.
+
+`storage` of an account is a key/value dictionary, and its values are
+integers within range of `[0, 2**256 - 1]`.
+
+`txs` are the steps which transform the pre-state into the post-state and
+must perform specific actions within the accounts (smart contracts) that result
+in verifiable changes to the balance, nonce, and/or storage in each of them.
+
+`post` is compared against the outcome of the client after the execution
+of each transaction, and any differences are considered a failure
+
+When designing a test, all the changes must be ideally saved into the contract's
+storage to be able to verify them in the post-state.
+
+### Test Transactions
+
+Transactions can be crafted by sending them with specific `data` or to a
+specific account, which contains the code to be executed
+
+Transactions can also create more accounts, by setting the `to` field to an 
+empty string.
+
+Transactions can be designed to fail, and a verification must be made that the
+transaction fails with the specific error that matches what is expected by the
+test.
+
+### Writing code for the accounts in the test
+
+Account bytecode can be embedded in the test accounts by adding it to the `code`
+field of the `account` object, or the `data` field of the `tx` object if the
+bytecode is meant to be treated as init code or call data.
+
+The code can be in either of the following formats:
+- `bytes` object, representing the raw opcodes in binary format
+- `str`, representing an hexadecimal format of the opcodes
+- `Code` compilable object
+
+Currently supported built-in compilable objects are:
+
+- `Yul` object containing [Yul source code][yul]
+
+`Code` objects can be concatenated together by using the `+` operator.
+
+### Verifying correctness of the new test
+
+A well written test performs a single verification output at a time.
+
+A verification output can be a single storage slot, the balance of an account,
+or a newly created contract.
+
+A test can be written as a negative verification. E.g. a contract is not
+created, or a transaction fails to execute or runs out of gas.
+
+These verifications must be carefully crafted because it is possible to end up
+having a false positive result, which means that the test passed but the
+intended verification was never made.
+
+To avoid these scenarios, it is important to have a separate verification to
+check that test is effective. E.g. when a transaction is supposed to fail, it
+is necessary to check that the failure error is actually the one expected by
+the test.
+
 [t8n]: https://github.com/ethereum/go-ethereum/tree/master/cmd/evm
 [b11r]: https://github.com/ethereum/go-ethereum/pull/23843
+[yul]: https://docs.soliditylang.org/en/latest/yul.html

setup.cfg

@@ -49,7 +49,7 @@ test =
 lint =
     isort>=5.8,<6
     mypy==0.910; implementation_name == "cpython"
-    black==21.5b2; implementation_name == "cpython"
+    black==22.3.0; implementation_name == "cpython"
     flake8-spellcheck>=0.24,<0.25
     flake8-docstrings>=1.6,<2
     flake8>=3.9,<4

src/ethereum_test/__init__.py

@@ -2,18 +2,27 @@
 Library for generating cross-client Ethereum tests.
 """
 
+from .code import Code
+from .common import TestAddress
 from .decorators import test_from, test_only
 from .fill import fill_state_test
+from .helpers import to_address
 from .state_test import StateTest
-from .types import Account, Environment, JSONEncoder, Transaction
+from .types import Account, Environment, JSONEncoder, Storage, Transaction
+from .yul import Yul
 
 __all__ = (
     "Account",
+    "Code",
     "Environment",
     "JSONEncoder",
     "StateTest",
+    "Storage",
+    "TestAddress",
     "Transaction",
+    "Yul",
     "fill_state_test",
     "test_from",
     "test_only",
+    "to_address",
 )

src/ethereum_test/code.py

@@ -2,15 +2,93 @@
 Code object that is an interface to different
 assembler/compiler backends.
 """
+from re import sub
+from typing import Union
 
 
 class Code(str):
     """
     Generic code object.
     """
 
+    bytecode: Union[bytes, None] = None
+
+    def __init__(self, code: Union[bytes, str, None]):
+        if code is not None:
+            if type(code) is bytes:
+                self.bytecode = code
+            elif type(code) is str:
+                if code.startswith("0x"):
+                    code = code[2:]
+                self.bytecode = bytes.fromhex(code)
+            else:
+                raise TypeError("code has invalid type")
+
     def assemble(self) -> bytes:
         """
         Assembles using `eas`.
         """
+        if self.bytecode is None:
+            return bytes()
+        else:
+            return self.bytecode
+
+    def __add__(self, other: Union[str, bytes, "Code"]) -> "Code":
+        """
+        Adds two code objects together, by converting both to bytes first.
+        """
+        return Code(code_to_bytes(self) + code_to_bytes(other))
+
+    def __radd__(self, other: Union[str, bytes, "Code"]) -> "Code":
+        """
+        Adds two code objects together, by converting both to bytes first.
+        """
+        return Code(code_to_bytes(other) + code_to_bytes(self))
+
+
+def code_to_bytes(code: Union[str, bytes, Code, None]) -> bytes:
+    """
+    Converts multiple types into bytecode.
+    """
+    if code is None:
         return bytes()
+
+    if isinstance(code, Code):
+        return code.assemble()
+
+    if type(code) is bytes:
+        return code
+
+    if type(code) is str:
+        # We can have a hex representation of bytecode with spaces for
+        # readability
+        code = sub(r"\s+", "", code)
+        if code.startswith("0x"):
+            return bytes.fromhex(code[2:])
+        return bytes.fromhex(code)
+
+    raise Exception("invalid type for `code`")
+
+
+def code_to_hex(code: Union[str, bytes, Code, None]) -> str:
+    """
+    Converts multiple types into a bytecode hex string.
+    """
+    if code is None:
+        return "0x"
+
+    if isinstance(code, Code):
+        return "0x" + code.assemble().hex()
+
+    if type(code) is bytes:
+        return "0x" + code.hex()
+
+    if type(code) is str:
+        # We can have a hex representation of bytecode with spaces for
+        # readability
+        code = sub(r"\s+", "", code)
+        if code.startswith("0x"):
+            return code
+        return "0x" + code
+
+    raise Exception("invalid type for `code`")