Kusama CC3 node sync stuck

[dryruner]

• It would help if you submit info about the system you are running, e.g.: operating system, kernel version, amount of available memory and swap, etc.

OS:
Ubuntu 18.04.4 LTS

Node Version:
polkadot 0.7.20-37381589-x86_64-linux-gnu

• Logs:

2020-02-24 03:14:40 Idle (1 peers), best: #777936 (0xc605…cc10), finalized #772475 (0x3096…0dee), ⬇ 2.4kiB/s ⬆ 2.6kiB/s
2020-02-24 03:14:42 Unable to author block in slot 263752347,. `can_author_with` returned: Failed to get runtime version at `BlockId::Hash(0xc6051b824a0ae1370dfe4198085c04babbc36d9c8bfcb57a9e3e96a3681bcc10)` and will disable authoring. Error: UnknownBlock("State already discarded for BlockId::Hash(0xc6051b824a0ae1370dfe4198085c04babbc36d9c8bfcb57a9e3e96a3681bcc10)") Probably a node update is required!

• Describe the role your node plays, e.g. validator, sentry, full node or light client.
  Full node, validator

• Any command-line options were passed?
  ./target/release/polkadot --validator

[dryruner]

From Telemetry it looks like v0.7.19 and v0.7.20 should work and sync up to date, so I don't really understand why the error says "Probably a node update is required!"

What should I do?

[svenski123]

I have repeatedly tried syncing up a full archive node over the last two days and every time the sync hangs on block 777936, typical log output below.

2020-02-24 15:05:17 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0xc6051b824a0ae1370dfe4198085c04babbc36d9c8bfcb57a9e3e96a3681bcc10)
2020-02-24 15:05:18 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0xc6051b824a0ae1370dfe4198085c04babbc36d9c8bfcb57a9e3e96a3681bcc10)
2020-02-24 15:05:19 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0xc6051b824a0ae1370dfe4198085c04babbc36d9c8bfcb57a9e3e96a3681bcc10)
2020-02-24 15:05:20 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0xc6051b824a0ae1370dfe4198085c04babbc36d9c8bfcb57a9e3e96a3681bcc10)
2020-02-24 15:05:20 Syncing, target=#1179186 (2 peers), best: #777936 (0xc605…cc10), finalized #775972 (0x51eb…85c3), ⬇ 120.5kiB/s ⬆ 5.8kiB/s
2020-02-24 15:05:21 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0xc6051b824a0ae1370dfe4198085c04babbc36d9c8bfcb57a9e3e96a3681bcc10)
2020-02-24 15:05:22 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0xc6051b824a0ae1370dfe4198085c04babbc36d9c8bfcb57a9e3e96a3681bcc10)
2020-02-24 15:05:23 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0xc6051b824a0ae1370dfe4198085c04babbc36d9c8bfcb57a9e3e96a3681bcc10)
2020-02-24 15:05:24 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0xc6051b824a0ae1370dfe4198085c04babbc36d9c8bfcb57a9e3e96a3681bcc10)

OS is Centos 7, polkadot built off of v0.7.20 github tag, rustc 1.41.0 (5e1a79984 2020-01-27).
Polkadot invoked with --pruning=archive option as archive mode is requried for sentry/validator.
I'm not sure if this issue occurs with standard archive pruning.

Block 0xc6051b82... is visible in polkascan here:
https://polkascan.io/pre/kusama/block/777936

[joepetrowski]

Also getting stuck on this block trying to sync an archive node.

• v0.7.20
• --execution Wasm --wasm-execution Compiled --pruning archive

When it hits this block, it loses the correct target block and starts dropping peers.

2020-02-25 09:20:22 Next epoch starts at slot 263313879
2020-02-25 09:20:24 Syncing 170.5 bps, target=#1189364 (25 peers), best: #764536 (0x7abe…7e59), finalized #764416 (0xedf6…3930), ⬇ 33.1kiB/s ⬆ 1.6kiB/s
2020-02-25 09:20:26 New epoch 1367 launching at block 0xe59d…c11a (block slot 263313880 >= start slot 263313879).
2020-02-25 09:20:26 Next epoch starts at slot 263314479
2020-02-25 09:20:29 Syncing 141.5 bps, target=#1189365 (25 peers), best: #765244 (0x6657…ff40), finalized #764928 (0xeab6…030b), ⬇ 214.9kiB/s ⬆ 2.7kiB/s
2020-02-25 09:20:30 New epoch 1368 launching at block 0x31c7…ebfd (block slot 263314479 >= start slot 263314479).
2020-02-25 09:20:30 Next epoch starts at slot 263315079
2020-02-25 09:20:30 Applying authority set change scheduled at block #765490
2020-02-25 09:20:30 Applying GRANDPA set change to new set with 160 authorities
2020-02-25 09:20:30 Imported justification for block #765490 that triggers command Changing authorities, signaling voter.
2020-02-25 09:20:34 Syncing 156.1 bps, target=#1189366 (25 peers), best: #766025 (0xe0cb…fb4d), finalized #765952 (0x6061…25ae), ⬇ 89.9kiB/s ⬆ 1.3kiB/s
2020-02-25 09:20:34 New epoch 1369 launching at block 0x2796…efea (block slot 263315079 >= start slot 263315079).
2020-02-25 09:20:34 Next epoch starts at slot 263315679
2020-02-25 09:20:37 New epoch 1370 launching at block 0xe2c2…3b9c (block slot 263315679 >= start slot 263315679).
2020-02-25 09:20:37 Next epoch starts at slot 263316279
2020-02-25 09:20:39 Syncing 164.3 bps, target=#1189367 (25 peers), best: #766848 (0xe3ec…cac6), finalized #766464 (0x01d7…486f), ⬇ 115.5kiB/s ⬆ 4.0kiB/s
2020-02-25 09:20:41 New epoch 1371 launching at block 0xe75a…4909 (block slot 263316279 >= start slot 263316279).
2020-02-25 09:20:41 Next epoch starts at slot 263316879
2020-02-25 09:20:42 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0x45c74c3a6fdc11353f75c13dc5d992c86be17b548e01d75979e45b65af8d0e24)
2020-02-25 09:20:42 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0x45c74c3a6fdc11353f75c13dc5d992c86be17b548e01d75979e45b65af8d0e24)
2020-02-25 09:20:43 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0x45c74c3a6fdc11353f75c13dc5d992c86be17b548e01d75979e45b65af8d0e24)
2020-02-25 09:20:43 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0x45c74c3a6fdc11353f75c13dc5d992c86be17b548e01d75979e45b65af8d0e24)
2020-02-25 09:20:43 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0x45c74c3a6fdc11353f75c13dc5d992c86be17b548e01d75979e45b65af8d0e24)
2020-02-25 09:20:43 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0x45c74c3a6fdc11353f75c13dc5d992c86be17b548e01d75979e45b65af8d0e24)
2020-02-25 09:20:44 Syncing 127.7 bps, target=#1189368 (19 peers), best: #767488 (0x45c7…0e24), finalized #767488 (0x45c7…0e24), ⬇ 327.3kiB/s ⬆ 2.5kiB/s
2020-02-25 09:20:44 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0x45c74c3a6fdc11353f75c13dc5d992c86be17b548e01d75979e45b65af8d0e24)
2020-02-25 09:20:44 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0x45c74c3a6fdc11353f75c13dc5d992c86be17b548e01d75979e45b65af8d0e24)
2020-02-25 09:20:45 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0x45c74c3a6fdc11353f75c13dc5d992c86be17b548e01d75979e45b65af8d0e24)
2020-02-25 09:20:45 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0x45c74c3a6fdc11353f75c13dc5d992c86be17b548e01d75979e45b65af8d0e24)
2020-02-25 09:20:46 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0x45c74c3a6fdc11353f75c13dc5d992c86be17b548e01d75979e45b65af8d0e24)
2020-02-25 09:20:47 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0x45c74c3a6fdc11353f75c13dc5d992c86be17b548e01d75979e45b65af8d0e24)
2020-02-25 09:20:47 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0x45c74c3a6fdc11353f75c13dc5d992c86be17b548e01d75979e45b65af8d0e24)
2020-02-25 09:20:48 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0x45c74c3a6fdc11353f75c13dc5d992c86be17b548e01d75979e45b65af8d0e24)
2020-02-25 09:20:49 Syncing  0.0 bps, target=#1189369 (13 peers), best: #767488 (0x45c7…0e24), finalized #767488 (0x45c7…0e24), ⬇ 1.1MiB/s ⬆ 13.5kiB/s
2020-02-25 09:20:49 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0x45c74c3a6fdc11353f75c13dc5d992c86be17b548e01d75979e45b65af8d0e24)
2020-02-25 09:20:49 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0x45c74c3a6fdc11353f75c13dc5d992c86be17b548e01d75979e45b65af8d0e24)
2020-02-25 09:20:52 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0x45c74c3a6fdc11353f75c13dc5d992c86be17b548e01d75979e45b65af8d0e24)
2020-02-25 09:20:53 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0x6088f657e6921bc6044387c340b29ec7db87567bea55ec693b9b10a2f26bcdf4)
2020-02-25 09:20:54 Syncing 25.5 bps, target=#1189369 (10 peers), best: #767616 (0x6088…cdf4), finalized #767488 (0x45c7…0e24), ⬇ 455.4kiB/s ⬆ 48.8kiB/s
2020-02-25 09:20:54 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0x6088f657e6921bc6044387c340b29ec7db87567bea55ec693b9b10a2f26bcdf4)
2020-02-25 09:20:55 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0x6088f657e6921bc6044387c340b29ec7db87567bea55ec693b9b10a2f26bcdf4)
2020-02-25 09:20:58 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0x6088f657e6921bc6044387c340b29ec7db87567bea55ec693b9b10a2f26bcdf4)
2020-02-25 09:20:58 Discovered new external address for our node: /ip4/10.0.1.134/tcp/30333/p2p/QmSVJo9Aik66CpM2bFfuEtSSCC13BoFbZ75eVkubtoxAQB
2020-02-25 09:20:59 Syncing  0.0 bps, target=#1189370 (7 peers), best: #767616 (0x6088…cdf4), finalized #767488 (0x45c7…0e24), ⬇ 269.3kiB/s ⬆ 20.6kiB/s
2020-02-25 09:20:59 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0x6088f657e6921bc6044387c340b29ec7db87567bea55ec693b9b10a2f26bcdf4)
2020-02-25 09:21:01 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0xac4dc8fccb1244214124ae06c3182a83c5482ea9388681ad758d4ca3845226ce)
2020-02-25 09:21:02 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0xac4dc8fccb1244214124ae06c3182a83c5482ea9388681ad758d4ca3845226ce)
2020-02-25 09:21:03 New epoch 1372 launching at block 0x2ba0…e198 (block slot 263316879 >= start slot 263316879).
2020-02-25 09:21:03 Next epoch starts at slot 263317479
2020-02-25 09:21:04 Syncing 51.1 bps, target=#777936 (4 peers), best: #767872 (0xf7d8…25dc), finalized #767488 (0x45c7…0e24), ⬇ 124.2kiB/s ⬆ 13.1kiB/s
2020-02-25 09:21:07 New epoch 1373 launching at block 0x6681…fc3c (block slot 263317481 >= start slot 263317479).
2020-02-25 09:21:07 Next epoch starts at slot 263318079
2020-02-25 09:21:09 Syncing 153.4 bps, target=#777936 (4 peers), best: #768640 (0xc462…d303), finalized #768512 (0x2e98…c6b1), ⬇ 77.5kiB/s ⬆ 81.3kiB/s
2020-02-25 09:21:10 New epoch 1374 launching at block 0x0d2c…753a (block slot 263318079 >= start slot 263318079).
2020-02-25 09:21:10 Next epoch starts at slot 263318679
2020-02-25 09:21:10 Applying authority set change scheduled at block #768963
2020-02-25 09:21:10 Applying GRANDPA set change to new set with 160 authorities
2020-02-25 09:21:10 Imported justification for block #768963 that triggers command Changing authorities, signaling voter.
2020-02-25 09:21:14 Syncing 178.9 bps, target=#777936 (4 peers), best: #769536 (0x4b59…43e7), finalized #769024 (0x64da…e685), ⬇ 62.5kiB/s ⬆ 10.2kiB/s
2020-02-25 09:21:14 New epoch 1375 launching at block 0x3fae…57d7 (block slot 263318679 >= start slot 263318679).
2020-02-25 09:21:14 Next epoch starts at slot 263319279
2020-02-25 09:21:17 New epoch 1376 launching at block 0x5ce2…156a (block slot 263319279 >= start slot 263319279).
2020-02-25 09:21:17 Next epoch starts at slot 263319879
2020-02-25 09:21:17 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0xd5f949471b701d952049c73ad2c7f2dfd9b92af323bb88902b75bb351bf86cf0)
2020-02-25 09:21:19 Syncing 127.9 bps, target=#777936 (4 peers), best: #770176 (0xd5f9…6cf0), finalized #769024 (0x64da…e685), ⬇ 102.4kiB/s ⬆ 13.6kiB/s
2020-02-25 09:21:21 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0xb57a71016a88b8e093221d0768bf928e4f92e224e82ce9223ded00b894c22984)
2020-02-25 09:21:22 New epoch 1377 launching at block 0x87e9…6edc (block slot 263319879 >= start slot 263319879).
2020-02-25 09:21:22 Next epoch starts at slot 263320479
2020-02-25 09:21:22 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0xb07dc732e69ecbac5bdadaba937dac040554c4f105e442f0da4bfae66b4fe833)
2020-02-25 09:21:24 Syncing 127.9 bps, target=#777936 (4 peers), best: #770816 (0xb07d…e833), finalized #769024 (0x64da…e685), ⬇ 78.0kiB/s ⬆ 10.8kiB/s
2020-02-25 09:21:26 New epoch 1378 launching at block 0xd7cd…9ca0 (block slot 263320479 >= start slot 263320479).
2020-02-25 09:21:26 Next epoch starts at slot 263321079
2020-02-25 09:21:29 Syncing 204.5 bps, target=#777936 (4 peers), best: #771840 (0x791d…f84d), finalized #769024 (0x64da…e685), ⬇ 56.7kiB/s ⬆ 10.4kiB/s
2020-02-25 09:21:30 New epoch 1379 launching at block 0x71cf…0227 (block slot 263321079 >= start slot 263321079).
2020-02-25 09:21:30 Next epoch starts at slot 263321679
2020-02-25 09:21:33 New epoch 1380 launching at block 0x3096…0dee (block slot 263321679 >= start slot 263321679).
2020-02-25 09:21:33 Next epoch starts at slot 263322279
2020-02-25 09:21:33 Applying authority set change scheduled at block #772475
2020-02-25 09:21:33 Applying GRANDPA set change to new set with 160 authorities
2020-02-25 09:21:33 Imported justification for block #772475 that triggers command Changing authorities, signaling voter.
2020-02-25 09:21:34 Syncing 127.9 bps, target=#1189376 (5 peers), best: #772480 (0x872d…7ecb), finalized #772475 (0x3096…0dee), ⬇ 60.1kiB/s ⬆ 18.5kiB/s
2020-02-25 09:21:34 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0x872d5e4c26b0fc53f4b2e419f7eb15c5b07334bd1277b1321a30263f8ab37ecb)
2020-02-25 09:21:39 Syncing 102.2 bps, target=#1189377 (5 peers), best: #772992 (0x302c…eb13), finalized #772608 (0xe825…1f36), ⬇ 79.8kiB/s ⬆ 17.1kiB/s
2020-02-25 09:21:39 New epoch 1381 launching at block 0xce28…c240 (block slot 263322279 >= start slot 263322279).
2020-02-25 09:21:39 Next epoch starts at slot 263322879
2020-02-25 09:21:39 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0x1ebaea73fe55825b8e13be32782294cd04ca77f6a172531fbbd43d02a4f08f11)
2020-02-25 09:21:43 New epoch 1382 launching at block 0xbe9e…d666 (block slot 263322879 >= start slot 263322879).
2020-02-25 09:21:43 Next epoch starts at slot 263323479
2020-02-25 09:21:44 Syncing 153.4 bps, target=#777936 (4 peers), best: #773760 (0xed4f…7979), finalized #773120 (0x1eba…8f11), ⬇ 77.7kiB/s ⬆ 7.7kiB/s
2020-02-25 09:21:46 New epoch 1383 launching at block 0x041a…aadb (block slot 263323479 >= start slot 263323479).
2020-02-25 09:21:46 Next epoch starts at slot 263324079
2020-02-25 09:21:49 Syncing 204.5 bps, target=#1189378 (6 peers), best: #774784 (0xa624…91fe), finalized #774144 (0x5687…26bd), ⬇ 65.8kiB/s ⬆ 8.2kiB/s
2020-02-25 09:21:49 New epoch 1384 launching at block 0x2280…4ee4 (block slot 263324079 >= start slot 263324079).
2020-02-25 09:21:49 Next epoch starts at slot 263324679
2020-02-25 09:21:49 Verification failed from peer: UnknownBlock: State already discarded for BlockId::Hash(0x893682d4e5e53a1982c6a742237d852dc90e5f8f0ed8da295a280af032ddd545)
2020-02-25 09:21:51 New epoch 1385 launching at block 0xd488…1116 (block slot 263324679 >= start slot 263324679).