Skip to content

Tracking: Hot/Cold storage separation #18682

@shekhirin

Description

@shekhirin

Describe the feature

Currently, Reth uses two storage systems for all data: MDBX database for hot data, and Nippy Jar static files for cold data. It works the following way:

  1. When new block is received, executed, and validated, we insert headers, transactions, and receipts into both database and static files.
  2. When reading headers/transactions/receipts, we look in both database and static files, in case if the data is present only in database.
  3. Pruner deletes the data from database, leaving it only in static files.

This doesn't make much sense now, but before we didn't write to static files directly, and instead were writing only to database, then StaticFileProducer was copying the data to static files, and Pruner removing it from database. Since the time we started writing to static files immediately on committing new block, we can simplify the whole storage system a lot.

The main goals of this tracking issue are:

  1. Do not ever have duplicate data between database and static files. This simplifies the mental model of storage, and allows to look into only one storage in providers.
  2. Move all append-only data to static files. This leaves only hot ever-changing data and indices in the database, making the access to historical data faster and the size smaller.

Additional context

No response

Sub-issues

Metadata

Metadata

Assignees

Labels

A-dbRelated to the databaseA-static-filesRelated to static filesC-debtA clean up/refactor of existing codeC-enhancementNew feature or requestC-tracking-issueAn issue that collects information about a broad development initiative

Type

No type

Projects

Status

In Progress

Relationships

None yet

Development

No branches or pull requests

Issue actions