Nodes using the wrong bootnode address

[Maharacha]

We have this bootnode address set since a few versions back:
/dns/polkadot-boot.dwellir.com/tcp/30334/ws/p2p/12D3KooWKvdDyRKqUfSAaUCbYiLwKY8uK3wDWpCuy2FiDLbkPTDJ
https://raw.githubusercontent.com/paritytech/polkadot/v0.9.43/node/service/chain-specs/polkadot.json

However, existing nodes running 0.9.43 are reporting:

The bootnode you want to connect to at `/dns/polkadot-boot.dwellir.com/tcp/30333/ws/p2p/12D3KooWKvdDyRKqUfSAaUCbYiLwKY8uK3wDWpCuy2FiDLbkPTDJ` provided a different peer ID `12D3KooWFHkJaz1BxqdAjzeLK4xcubxXP25kTaYWZqsJukuex5Ly` than the one you expect `12D3KooWKvdDyRKqUfSAaUCbYiLwKY8uK3wDWpCuy2FiDLbkPTDJ`.

I can't reproduce it with new nodes. Running:

./polkadot --chain polkadot -ldebug 2>&1 | tee polkadot.log

Shows correct:

libp2p\_swarm: Connection established: PeerId("12D3KooWKvdDyRKqUfSAaUCbYiLwKY8uK3wDWpCuy2FiDLbkPTDJ") Dialer { address: "/dns/polkadot-boot.dwellir.com/tcp/30334/ws/p2p/12D3KooWKvdDyRKqUfSAaUCbYiLwKY8uK3wDWpCuy2FiDLbkPTDJ", role\_override: Dialer }; Total (peer): 1.

We are running the bootnode on 0.9.43 with these cli args:

"--chain polkadot --prometheus-external --rpc-port=9933"

Hence, using default p2p port 30333 on the node but forwards from port 30334 in the firewall.

How can this happen? Is it not that the nodes are exclusively using the addresses in the polkadot.json file?

[bkchr]

However, existing nodes running 0.9.43 are reporting:

I can't reproduce it with new nodes. Running:

I don't get what the difference is between these two?

[Maharacha]

However, existing nodes running 0.9.43 are reporting:

I can't reproduce it with new nodes. Running:

I don't get what the difference is between these two?

I'm not sure there is. It might just be that the nodes are trying to connect to this wrong address every now and then. But when you start the node it will also connect to the correct address. I guess the question is:
Can the bootnode address be announced from somewhere else than from the polkadot.json file?
Like in our case, the bootnode is actually listening on port 30333 but the address in polkadot.json is on 30334 and we forward 30334 to 30333 in the firewall. If the node announce on the p2p network that it is listening on 30333, could this make other nodes believe this bootnode is available on port 30333?
If this is the case, I guess the solution would be to avoid using different ports in the NAT:ing.

[Maharacha]

I have set --port 30334 and made the NAT rule to forward to that instead. Let's see if it helps. Will probably know by tomorrow since the error appears a few times a day.

[bkchr]

If the node announce on the p2p network that it is listening on 30333, could this make other nodes believe this bootnode is available on port 30333?

Ahh yeah, now I get what the error is. Each node will learn from other nodes it is connected to which "address" they see from them. If some node was observing your node with the 30333 port, it reported this to the node. The node then gossiped these information and at some point some tried to connect using this address.

[Maharacha]

If the node announce on the p2p network that it is listening on 30333, could this make other nodes believe this bootnode is available on port 30333?

Ahh yeah, now I get what the error is. Each node will learn from other nodes it is connected to which "address" they see from them. If some node was observing your node with the 30333 port, it reported this to the node. The node then gossiped these information and at some point some tried to connect using this address.

I see. I guess the error is kind of false then. I mean the bootnode actually works for nodes using the information in polkadot.json. But then, they are doing regular checks based on this gossiping, and starts to use the wrong address. Would it not make more sense to only use the information from polkadot.json when connecting to bootnodes?