Hybrid Consensus Node #1876

liamaharon · 2025-07-22T09:39:10Z

Phase 1 in #1887.

I added the full NPoS dependencies in this PR primarily to reduce future merge conflicts, which recently been a bit time consuming to resolve.

Hybrid Node

The Hybrid Node handles producing, syncing and validating both Aura and Babe based blocks. It contains logic to dynamically switch between Aura and Babe based processing, depending on the specific block it needs to process.

Implementation

Node Service

The core changes are in the node service logic. Consensus specific service logic was migrated to a new ConsensusMechanism trait, which I have written both Aura and Babe implementation for.

Node RPC

Node rpc logic was made generic across consensus mechanisms by allowing the caller to pass the Frontier Consensus Data Provider to use, and custom RPC modules to instantiate.

CLI / Command

A new cli parameter --initial-consensus was added, which determines which ConsensusMechanism to first start the node in. The node will dynamically switch between ConsensusMechanisms at runtime as needed to continue block production / validation / import.

TODO

Fix compiling with runtime-benchmarks feature flag
Check ConditionalEVMBlockImport is up to date

shamil-gadelshin

An interesting approach, you've created the consensus engine abstraction on top of the existing crates! I have several questions about it:

Do you have the whole roadmap of the consensus change? It would be easier to know what part of the code is missing and what we expect in the next parts.
How do you test the consensus change? It seems it's not possible until the next part is merged.
Am I correct that if I propagate the custom babe block I can switch the consensus from a random node?
What do you think about the specific block number as an explicit consensus engine boundary? This way we'll get rid of the pending verification, string error as a marker, etc.

node/src/cli.rs

node/src/command.rs

node/src/consensus_mechanism.rs

node/src/command.rs

node/src/consensus/aura_wrapped_import_queue.rs

node/src/consensus/babe_consensus.rs

node/src/command.rs

shamil-gadelshin · 2025-07-24T12:30:29Z

node/src/consensus/aura_consensus.rs

+            loop {
+                // Check if the runtime is Babe once per block.
+                if let Ok(c) = sc_consensus_babe::configuration(&*client) {
+                    if !c.authorities.is_empty() {


Can we hardcode the exact block number to switch?

I prefer to reduce "magic numbers" that have the possibility to be misconfigured where possible. However if there is benefit to switching on block number instead of this I'm open to it

liamaharon · 2025-07-25T05:26:57Z

Thank you @shamil-gadelshin for your detailed review!

Do you have the whole roadmap of the consensus change? It would be easier to know what part of the code is missing and what we expect in the next parts.

Yes, you can view the full changes (including runtime upgrade + migration) here: #1708

tldr; next changes after merging this are

enacting a runtime upgrade migrating existing PoA authorities to NPoS authorities, and do not allow changes in that set
slowly adding new whitelisted parties into that set
opening the set up to anyone to validate

How do you test the consensus change? It seems it's not possible until the next part is merged.

You can test them on the node-decentralization branch, following the same steps I did here: https://discord.com/channels/1120750674595024897/1387662637843742760/1391994494714515588

If you're interested in it let me know and I'll send you exact commands I used to set it up :)

Am I correct that if I propagate the custom babe block I can switch the consensus from a random node?

This is an interesting edge case I have not considered! I think you are correct, I would need to do additional checks on the Babe block rather than switching straight to Babe consensus...

What do you think about the specific block number as an explicit consensus engine boundary? This way we'll get rid of the pending verification, string error as a marker, etc.

Yeah this is something I considered. Initially I tried to keep away from needing to hardcode any block numbers, keeping this as close as I could to being a "normal" runtime upgrade. However you raise good points about verification. I'll have a think about it.

shamil-gadelshin

Thanks for the changes and the roadmap. It's easier to understand the general direction now.

My current concern is the moment of the consensus change. We don't need flexibility for this event because it seems like "one time unidirectional change". If the hardcoded magic const makes the code simpler, let's use it. If it introduces more issues than it solves, let's use your current method (presence of babe authorities) or a similar external marker.

node/src/cli.rs

node/src/consensus_mechanism.rs

shamil-gadelshin · 2025-07-31T11:35:14Z

2025-07-28 10:51:15 panicked at ./subtensor/runtime/src/lib.rs:2421:13:
not implemented    
2025-07-28 10:51:15 💤 Idle (2 peers), best: #10 (0x85b3…3ec4), finalized #8 (0x7206…08d9), ⬇ 1.3kiB/s ⬆ 1.7kiB/s    
2025-07-28 10:51:16 panicked at ./subtensor/runtime/src/lib.rs:2421:13:
not implemented    
2025-07-28 10:51:16 💤 Idle (2 peers), best: #10 (0x85b3…3ec4), finalized #8 (0x7206…08d9), ⬇ 1.6kiB/s ⬆ 1.4kiB/s    
2025-07-28 10:51:16 panicked at ./subtensor/runtime/src/lib.rs:2421:13:
not implemented

@gztensor saw this error after the runtime upgrade. It seems that "babe_switch" worker asks BabeApi and gets "not implemented"

Do we have a fix for this one?

liamaharon · 2025-08-01T04:14:49Z

@gztensor saw this error after the runtime upgrade. It seems that "babe_switch" worker asks BabeApi and gets "not implemented"

Do we have a fix for this one?

@shamil-gadelshin thanks for giving it another try! It will panic if you try to use the Babe node with an Aura runtime. This branch does not include the change to the new Babe NPoS runtime, which I suspect is why you see that panic.

Please try following my demo here https://discord.com/channels/1120750674595024897/1387662637843742760/1399902628309106822 where I test it on the node-decentralization branch. That branch includes both the hybrid node changes from this branch, as well as the new Babe NPoS runtime.

Let me know how you go!

shamil-gadelshin

Runtime upgrade works without bugs. Warp sync works as expected.

sam0x17 · 2025-08-07T15:05:27Z

reverted until we can test on mainnet clone and a few other things

liamaharon added 4 commits July 22, 2025 18:31

hybrid node

348626f

comment

1a20aab

fix runtime-benchmark compile

c1e1fbe

update ConditionalEVMBlockImport

371761d

liamaharon marked this pull request as ready for review July 22, 2025 23:17

clippy

35f7ff2

liamaharon force-pushed the hybrid-node branch from 4f25b1c to 35f7ff2 Compare July 22, 2025 23:43

fix runtime-benchmark compile again

4789a88

shamil-gadelshin self-requested a review July 23, 2025 15:03

shamil-gadelshin requested changes Jul 24, 2025

View reviewed changes

liamaharon added 5 commits July 25, 2025 18:31

prefer matches!

db92646

cleanup unused file

4b3beab

prefer matches!

aec6b29

prefer matches!

5fd36bb

patch vulnerability in meantime while wait for psdk fork

adb9648

liamaharon mentioned this pull request Jul 27, 2025

[Meta] NPoS + Babe Roadmap #1887

Open

15 tasks

liamaharon changed the title ~~Hybrid Node~~ Hybrid Consensus Node Jul 27, 2025

liamaharon requested a review from shamil-gadelshin July 27, 2025 02:05

Merge branch 'devnet-ready' into hybrid-node

1359f3e

shamil-gadelshin reviewed Jul 28, 2025

View reviewed changes

node/src/cli.rs Show resolved Hide resolved

gztensor previously approved these changes Jul 28, 2025

View reviewed changes

node/src/consensus_mechanism.rs Outdated Show resolved Hide resolved

add todo to remove initial_consensus flag.

f055227

liamaharon dismissed gztensor’s stale review via f055227 July 31, 2025 01:33

liamaharon added 2 commits July 31, 2025 13:37

better fn names

5a63327

Merge branch 'devnet-ready' into hybrid-node

21e492f

liamaharon added the no-spec-version-bump PR does not contain changes that requires bumping the spec version label Jul 31, 2025

liamaharon requested review from shamil-gadelshin and gztensor July 31, 2025 01:56

liamaharon added 2 commits August 4, 2025 14:51

Return default values in BabeApi instead of panic prior to Babe upgrade

f8b783e

Merge branch 'devnet-ready' into hybrid-node

d900522

liamaharon force-pushed the hybrid-node branch from 89b7c38 to d900522 Compare August 4, 2025 02:56

Bump spec version

e886766

liamaharon removed the no-spec-version-bump PR does not contain changes that requires bumping the spec version label Aug 4, 2025

liamaharon added 3 commits August 5, 2025 15:48

merge devnet-ready

969a23e

merge

bda2df5

cleanup

f88f162

shamil-gadelshin previously approved these changes Aug 5, 2025

View reviewed changes

run zepter

7a00604

liamaharon dismissed shamil-gadelshin’s stale review via 7a00604 August 5, 2025 23:59

cargo +nightly fmt

43dea13

liamaharon requested a review from shamil-gadelshin August 6, 2025 00:00

shamil-gadelshin approved these changes Aug 6, 2025

View reviewed changes

gztensor approved these changes Aug 7, 2025

View reviewed changes

liamaharon merged commit 8febe38 into devnet-ready Aug 7, 2025
88 of 90 checks passed

sam0x17 mentioned this pull request Aug 7, 2025

Revert "Hybrid Consensus Node" #1924

Closed

liamaharon mentioned this pull request Aug 7, 2025

Node Support for Aura -> Babe Runtime Upgrades #1927

Merged

19 tasks

This was referenced Aug 12, 2025

Add back liam consensus stuff #1941

Merged

mainnet deploy 8/19/2025 #1958

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Hybrid Consensus Node #1876

Hybrid Consensus Node #1876

liamaharon commented Jul 22, 2025 •

edited

Loading

Uh oh!

shamil-gadelshin left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shamil-gadelshin Jul 24, 2025

Uh oh!

liamaharon Jul 25, 2025

Uh oh!

liamaharon commented Jul 25, 2025 •

edited

Loading

Uh oh!

shamil-gadelshin left a comment

Uh oh!

Uh oh!

Uh oh!

shamil-gadelshin commented Jul 31, 2025

Uh oh!

liamaharon commented Aug 1, 2025 •

edited

Loading

Uh oh!

shamil-gadelshin left a comment

Uh oh!

Uh oh!

sam0x17 commented Aug 7, 2025

Uh oh!

Uh oh!

Hybrid Consensus Node #1876

Hybrid Consensus Node #1876

Conversation

liamaharon commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Hybrid Node

Implementation

Node Service

Node RPC

CLI / Command

TODO

Uh oh!

shamil-gadelshin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shamil-gadelshin Jul 24, 2025

Choose a reason for hiding this comment

Uh oh!

liamaharon Jul 25, 2025

Choose a reason for hiding this comment

Uh oh!

liamaharon commented Jul 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shamil-gadelshin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

shamil-gadelshin commented Jul 31, 2025

Uh oh!

liamaharon commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shamil-gadelshin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sam0x17 commented Aug 7, 2025

Uh oh!

Uh oh!

liamaharon commented Jul 22, 2025 •

edited

Loading

liamaharon commented Jul 25, 2025 •

edited

Loading

liamaharon commented Aug 1, 2025 •

edited

Loading