296 fix blocks too big

[nischitpra]

Summary by CodeRabbit

• New Features

  • Added CLI commands: backfill and committer.
  • Environment-based configuration via .env and environment variables.
  • Exposes Prometheus metrics on port 2112 with expanded per-project/per-chain metrics.

• Improvements

  • Backfill writes Parquet files to S3 with dynamic batching and resiliency.
  • Committer streams processed data from S3 to downstream systems with memory-safe flow control.
  • Enhanced RPC handling for specific chains to improve block/transaction retrieval reliability.

• Chores

  • Ignore Parquet files in version control.
  • Updated dependencies and docker-compose Kafka settings.

[zeet-co]

We're building your pull request over on Zeet.
Click me for more info about your build and deployment.
Once built, this branch can be tested at: https://insight-296-hedera-testnet-backfi-d61367.insight.zeet.app before merging 😉

[coderabbitai]

Caution

Review failed

The pull request is closed.

✨ Finishing touches

• 📝 Generate Docstrings

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch np/296_fix

---

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

Disabled knowledge base sources:

• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between f66f782 and ef7e268.

⛔ Files ignored due to path filters (1)

• go.sum is excluded by !**/*.sum

📒 Files selected for processing (30)

• .gitignore (1 hunks)
• cmd/backfill.go (1 hunks)
• cmd/committer.go (1 hunks)
• cmd/migrate_valid.go (7 hunks)
• cmd/root.go (1 hunks)
• configs/config.go (3 hunks)
• docker-compose.yml (0 hunks)
• go.mod (2 hunks)
• internal/backfill/backfill.go (1 hunks)
• internal/backfill/disableIndexerMaybeStartCommitter.go (1 hunks)
• internal/backfill/getbackfillboundaries.go (1 hunks)
• internal/backfill/parquetwriter.go (1 hunks)
• internal/committer/blockparserroutine.go (1 hunks)
• internal/committer/blockprocessorroutine.go (1 hunks)
• internal/committer/committer.go (1 hunks)
• internal/committer/parquet.go (1 hunks)
• internal/committer/poollatest.go (1 hunks)
• internal/committer/semaphore.go (1 hunks)
• internal/libs/clickhouse.go (1 hunks)
• internal/libs/constants.go (1 hunks)
• internal/libs/kafka.go (1 hunks)
• internal/libs/libblockdata/getblockdata.go (1 hunks)
• internal/libs/libblockdata/validator.go (1 hunks)
• internal/libs/rpcclient.go (1 hunks)
• internal/libs/s3.go (1 hunks)
• internal/metrics/metrics.go (1 hunks)
• internal/rpc/rpc.go (3 hunks)
• internal/storage/kafka.go (2 hunks)
• internal/storage/kafka_publisher.go (4 hunks)
• internal/types/types.go (1 hunks)

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	github.com/​caarlos0/​env@​v3.5.0+incompatible
	github.com/​joho/​godotenv@​v1.5.1

View full report