Investigate cloning MDBX transaction in state provider builder

[shekhirin]

Summary

In StateProviderBuilder::build(), we currently create a new state provider by calling state_by_block_hash. MDBX supports cloning transactions via its API, which could potentially be leveraged here instead.

Context

https://github.com/paradigmxyz/reth/blob/main/crates/engine/tree/src/tree/mod.rs#L119-L122

pub fn build(&self) -> ProviderResult<StateProviderBox> {
    let mut provider = self.provider_factory.state_by_block_hash(self.historical)?;
    // ...
}

Task

• Investigate whether using the MDBX transaction clone API would be beneficial here
• Determine if cloning a transaction is cheaper than creating a new one via state_by_block_hash
• If beneficial, implement the change using the appropriate MDBX API

[figtracer]

gmmm can i work on this one?

[shekhirin]

@figtracer we'll do this one in-house, please check other issues!

[shekhirin]

cc @erthink, from what I read in the MDBX documentation, there's currently no way to clone a read transaction, or to create a child read transaction, so that we can guarantee the same snapshot is being used for all read transactions. Is that right?

[mediocregopher]

@erthink piggybacking on @shekhirin 's question: we see that this restriction is in place:

A transaction and its cursors must only be used by a single thread, and a thread may only have a single transaction at a time unless the MDBX_NOSTICKYTHREADS is used.

Does this apply to child/nested txs as well? Ie if we create a nested tx does it have to stay on the same thread as its parent? Or can we use nested txs as a way to sync read-only state across threads?

[erthink]

Each reading transaction uses some MVCC-snapshot of a data. Currently, at the start of each reading transaction, the last committed MVCC snapshot is selected without alternative. So two consecutive reading transactions can use different MVCC-snapshots if a writing transaction is committed when ones start.

Cloning of (non-nested) read transactions is not provided at the moment, simply because it makes little sense and so far no one has needed it enough to implement it. However, I do not rule out that this feature will be useful, as it allows you to parallelize reading operations between multiple threads without having to synchronize operations related to the transaction instance.
If you need such functionality, then I can add it to the queue (but only after improvements sponsored by Erigon).

There is also no support for nested reading transactions:

• It is completely pointless to create a child reading transaction for the reading parent, since ones will not be able to work in parallel, but will see exactly the same version of a data.
• For a writing parent transaction, creating a child reading transaction will only mean conditional control and prevention of operations modifying data.
  So far, this feature has not been implemented due to the need to complicate the logic of internal checks.
  Again, if you need such functionality, then I can add it to the queue (but only after improvements sponsored by Erigon).

The implementation of the first feature seems simpler, and most importantly less risky in terms of the occurrence of regressions in rare tricky conditions.

---

Regardless of the above, there are two basic rules regarding the use of transactions:

• an any transaction and its associated cursors MUST NOT be used simultaneously from different threads. NEVER!
  Otherwise, arbitrary poorly reproducible problems are inevitable, such as these Assertion failed: while closing cursor function panic file mdbx line 0. isar-community/isar-community#44, isar_community_flutter_libs 3.3.0-dev.3 EXC_BAD_ACCESS crashes. isar-community/isar-community#64, etc.
• parent transactions MUST NOT be used when there are child transactions (except for the simplest API calls not involving data and cursors).

[erthink]

Does this apply to child/nested txs as well? Ie if we create a nested tx does it have to stay on the same thread as its parent? Or can we use nested txs as a way to sync read-only state across threads?

AFAIK, Reth uses MDBX_NOSTICKYTHREADS. However, the above rule (and everything I said above) applies to this case as well:

• parent txns MUST NOT be used while there are child/nested ones;
• with MDBX_NOSTICKYTHREADS child/nested can be used from any thread, but not from several at the same time;
• w/o MDBX_NOSTICKYTHREADS child/nested can be used only from the same thread as the parent.

[erthink]

• If beneficial, implement the change using the appropriate MDBX API

Please show the C-prototype of the API function you want to get.

[mediocregopher]

Thanks @erthink that all makes sense and basically matches what we assumed the case to be. Just checking that we hadn't misundertood anything :)

it allows you to parallelize reading operations between multiple threads without having to synchronize operations related to the transaction instance

This is the exact use-case we're looking at. We have a situation where we want multiple threads working on the same view of the DB, and for them to synchronize that view can involve first fetching a non-trivial amount of data out of the db to use as an in-memory overlay. If we were able to instead distribute cloned read-only txs to each thread that would save a lot of IO in a critical path.

Please show the C-prototype of the API function you want to get.

Something like this is essentially what we're thinking:

mdbx_txn_clone(MDBX_txn * existing, MDBX_txn ** into)

Where existing is an active read-only transaction and into is being cloned into.

[erthink]

mdbx_txn_clone(MDBX_txn * existing, MDBX_txn ** into)

In general, this is not a problem, but in libmdbx it is customary to bother with performance and take measures to minimize the likelihood of incorrect API usage by users.

With this in mind, the proposed function has three drawbacks:

1. It does not allow reuse handles/instances of reading transactions after the mdbx_txn_reset().
2. A guarantee is required that the cloned read transaction will not be competitively aborted/reset by another thread or asynchronous handler.
3. When running in non-MDBX_NOSTICKYTHREADS mode, it is not possible to clone a transaction for use in another thread (cloning must be performed in the target thread, but then the original transaction cannot be used in it).

I'll think about how to do it better.