Light client finalized stuck at #28175 on Kusama chain

[amaury1093]

• It would help if you submit info about the system you are running, e.g.: operating system, kernel version, amount of available memory and swap, etc. Tried in macOS 10.15.2, and with the wasm binary
• Describe the role your node plays, e.g. validator, sentry, full node or light client. Light Client.
• Any command-line options were passed? --light

Repro:

• polkadot --light
• Wait for ~2h for it to sync to the tip

Expected:
Finalized block should match actual finalized

Actual:

[2020-01-23T16:59:01.069Z] info 2020-01-23 17:59:01 Imported #751213 (0xae26…ce92)

[2020-01-23T16:59:01.789Z] info 2020-01-23 17:59:01 Reorg from #751213,0xae26…ce92 to #751213,0xc566…ee6d, common ancestor #751212,0x7933…989f
2020-01-23 17:59:01 Imported #751213 (0xc566…ee6d)

[2020-01-23T16:59:04.645Z] info 2020-01-23 17:59:04 Idle (20 peers), best: #751213 (0xc566…ee6d), finalized #28175 (0x456c…19f0), ⬇ 8.9kiB/s ⬆ 5.6kiB/s

[2020-01-23T16:59:06.855Z] info 2020-01-23 17:59:06 Imported #751214 (0x4339…20ff)

[2020-01-23T16:59:09.649Z] info 2020-01-23 17:59:09 Idle (20 peers), best: #751214 (0x4339…20ff), finalized #28175 (0x456c…19f0), ⬇ 8.0kiB/s ⬆ 4.6kiB/s

[2020-01-23T16:59:13.243Z] info 2020-01-23 17:59:13 Imported #751215 (0x123b…fc6d)

[2020-01-23T16:59:14.654Z] info 2020-01-23 17:59:14 Idle (20 peers), best: #751215 (0x123b…fc6d), finalized #28175 (0x456c…19f0), ⬇ 8.5kiB/s ⬆ 6.2kiB/s

[2020-01-23T16:59:18.923Z] info 2020-01-23 17:59:18 Imported #751216 (0xdb1a…4a72)

[2020-01-23T16:59:19.661Z] info 2020-01-23 17:59:19 Idle (20 peers), best: #751216 (0xdb1a…4a72), finalized #28175 (0x456c…19f0), ⬇ 7.6kiB/s ⬆ 6.4kiB/s

[2020-01-23T16:59:20.835Z] info 2020-01-23 17:59:20 Discovered new external address for our node: /ip4/10.10.10.1/tcp/30333/p2p/QmUcJAJ7JZBpbYP2mRaAFm2pMRNpfVVDYkLiNHjrU2tzPd

[2020-01-23T16:59:24.542Z] info 2020-01-23 17:59:24 Imported #751217 (0xa209…53eb)

[2020-01-23T16:59:24.664Z] info 2020-01-23 17:59:24 Idle (20 peers), best: #751217 (0xa209…53eb), finalized #28175 (0x456c…19f0), ⬇ 7.7kiB/s ⬆ 6.5kiB/s

[2020-01-23T16:59:29.670Z] info 2020-01-23 17:59:29 Idle (20 peers), best: #751217 (0xa209…53eb), finalized #28175 (0x456c…19f0), ⬇ 6.4kiB/s ⬆ 5.5kiB/s

[rphmeier]

cc @svyatonik

[svyatonik]

I've been looking at this issue ~couple of weeks ago. And the cause was that we have switched from proof-of-call to proof-of-read for GRANDPA authorities sets (see #3734 for details - I've ever left comment about that somewhere inside). So all blocks up to (and including) 28175 are finalized by the first authorities set => we accept all generated justifications. Then there's authorities set change that enacts at this block => we stop accepting new justifications && ask for finality proof instead. Finality proof provider is trying to generate finality proof using proof-of-read and fails because at the time of that block there were no authorities at given storage cell. That's why we never actually get any proofs on light nodes => never finalize these blocks.

@svyatonik , @AmauryM , closable?