Light client - verification failed for requesting missing blocks

[Eligioo]

New issue checklist

• I have read all of the README
• I have searched existing issues and this is not a duplicate.

General information

• Library version(s):0.20
• Devices/Simulators/Machine affected: Light clients
• Reproducible in the testnet? (Yes/No): yes

Bug report

When a light client has to request missing blocks, there is a chance that the verification of this request fails. It could be that this issue exists for quite a while now but never really got noticed until in recent versions light clients had a low peer count (#2277) and not all gossipsub block header messages are received (#2288). Because of this combination, the node has to perform more missing block requests, probably causing the chance of this issue occurring to increase. When the verification fails the node is unable to continue accepting new blocks and can't recover from this state. Even when the macro sync kicks in.

It appears that blocks that already have been accepted, are requested as missing blocks. Also it tries to push it again from the missing block response.

Note block 17778523 and 17778525 in the following snippet. And gossipsubs of 17778524 and 17778526 to be out of order.

2024-03-04T08:28:31.900441000Z DEBUG push                 | Accepted block block=#17778522:MI:00949eb694 kind="extend"
2024-03-04T08:28:31.915955000Z DEBUG push                 | Accepted block block=#17778523:MI:a7cd50e34d kind="extend"
2024-03-04T08:28:33.573134000Z DEBUG block_queue          | Received block via gossipsub block=#17778526:MI:5711919193 peer_id=12D3KooWGbKfQvV4r59BCYHoRzGbARTRFsxbx5SpUQb8C3ciQSiM
2024-03-04T08:28:33.574778000Z DEBUG block_queue          | Requesting missing blocks block_number=17778525 block_hash=6b1e5d0e649d17e7112279614277c8dcb07aafb56807ee08fac9d4cb9733ec01 head_hash=a7cd50e34dcf087199386aac6a26a8753af7df72003ce1fa9c2f028a3651e314 macro_height=17778464
2024-03-04T08:28:33.586606000Z DEBUG block_queue          | Received block via gossipsub block=#17778524:MA:ea5dd050b2 peer_id=12D3KooWGbKfQvV4r59BCYHoRzGbARTRFsxbx5SpUQb8C3ciQSiM
2024-03-04T08:28:33.830005000Z DEBUG push                 | Accepted block block=#17778524:MA:ea5dd050b2 kind="extend"
2024-03-04T08:28:33.830403000Z DEBUG block_queue          | Received block via gossipsub block=#17778525:MI:6b1e5d0e64 peer_id=12D3KooWGbKfQvV4r59BCYHoRzGbARTRFsxbx5SpUQb8C3ciQSiM
2024-03-04T08:28:33.837976000Z DEBUG push                 | Accepted block block=#17778525:MI:6b1e5d0e64 kind="extend"
2024-03-04T08:28:33.845859000Z DEBUG push                 | Accepted block block=#17778526:MI:5711919193 kind="extend"
2024-03-04T08:28:34.126366000Z DEBUG block_queue          | Requesting missing blocks block_number=17778525 block_hash=6b1e5d0e649d17e7112279614277c8dcb07aafb56807ee08fac9d4cb9733ec01 head_hash=5711919193f2b22822c42a990200ad0adaba43737a5e378531cf56300339451f macro_height=17778524
2024-03-04T08:28:34.126695000Z DEBUG block_queue          | Requesting missing blocks block_number=17778523 block_hash=a7cd50e34dcf087199386aac6a26a8753af7df72003ce1fa9c2f028a3651e314 head_hash=5711919193f2b22822c42a990200ad0adaba43737a5e378531cf56300339451f macro_height=17778524
2024-03-04T08:28:34.209881000Z DEBUG queue                | Pushing block #17778525:MI:6b1e5d0e64 from missing blocks response
2024-03-04T08:28:34.210276000Z DEBUG block_request_compo… | Received empty missing blocks response
2024-03-04T08:28:34.210332000Z DEBUG sync_queue           | Verification failed peer_id=2 id=MissingBlockRequest { target_block_number: 17778523, target_block_hash: a7cd50e34dcf087199386aac6a26a8753af7df72003ce1fa9c2f028a3651e314, epoch_validators: Validators { validators: [Validator { address: Address("1dab7b1fc3483d8822eb223a3a84bf5a7a033081"), voting_key: LazyPublicKey(7c127cda41181fc2fff5ef49b38d5b4b791aa02bac26bbd227f8564a26d38eb2619901b8bb465c00f2fe68b15c3cf26e59dbaea8cb07882065827950442c052ceeb171ae3e454acbc347aca42fc71d14ced130218e7761169bc5a9fd740501ca774db39197e2bc4df5853c4e354fc3f4804e9a96b1cc33e2720b314097ccb553439b6b33446d10b7f76b1c7ea90882d56fd30c58d9454589ce1dea6b9e8295c9de1c8c0e5c6615a8c1c188948f0d5d4dffbc842ff97b50b2fc2a77f07000b3a58025fddca83aca027166c92469f3b84c88cbc894e7fed33bc70b1bedfaa47e1050196a228fde5e1dbec94e48a907b15a72d5f84ef6afc098e29fcd626a9d6eaff12eee1774d0f420b7dc5be943b8cd4afea715da95b90f63343e217300), signing_key: d419c718543cbeb6068e322800d53aeb7bc90f221fcc92c80247ccd914ced438, slots: 0..65 }, Validator { address: Address("6c6c822f592ff5e7b07d5acb284ea282c54884ec"), voting_key: LazyPublicKey(81f8c6e4d629b91fae9afb4fcb4ce5639002cc5a27b10669752cae00d60879133681dc113e4ea5acc05e1a157b2bf07ddcbfa449a8999f9420cc4fc57bd25fdb23d564a5d9de0337853415342090c17d5131fad9c1c81e8632410b803d3b00baf9bd234d3510512665f083a644eca7bbe41fc1c06c1cead3d4e4f52550182af411e0cff067bb749d23e3d22784be671902813062f04f7877c682dd6039f954caea0038704e91614b0389b6ae584e81526f16040297dceaf4ba994af5d100543d87de2dc0c033449d706b6a4dad44ea90f49e7fcbd888dec4e95cc35efaf04108335ef74f97730dd0c809b25a737f7f9e821800e27b2e3078b65aa4af516adc649a514f6cb46981f3bf080f312a872e3a120b709905310ff3b75fe06681), signing_key: b1881f7b5412d456a9acb7d9021d3b032686608c407b487ff377d07cd9fbf244, slots: 65..168 }, Validator { address: Address("7a493b7fa61f4ed323c67f94d25d95f5719bdc9c"), voting_key: LazyPublicKey(fcdd7c2a13dfa03fb29ea92c8edbd4315055d84114f75a3bc36fc311e6eae698e9cc5d6c7e1e4a4a0f937a694fa666e47eb6f1a21d44a4f9b69f2acf6f25e3289d77c3bb3caab154dee0e263ec79c0a263e5c84c45c7e2d84c8f7f96f356015be91276a5d6e4923009ccc42fff3474b51609f31dff4410b25017a265dc350c698a06597391fb67a15003df808da1cdf70aa313c80cae0e793c6289d7e420ab4084acf32d531ebf2eafbe73e6b2ab923f3589efc75ba8e9bb0c62a7e11e00116e3f1a4164540b01073decb05ca56c83ec6eacb4af0234910b1b0133f06556003601e5e00d25d905c68c9fb95e3c28859cadd49b23a2a1b5606d63de7b2f1b8806badef3fb188d9be4cb9215926762b6cf0743d44c8790544f0a77be2701), signing_key: ee1424f128e4b8ae9a4951b7fb4717fa94cd9c74f29dbffdf591cdedbe149a8d, slots: 168..231 }, Validator { address: Address("83cbb5be504fdde49fb2948476a7aa204933d205"), voting_key: LazyPublicKey(23dede6c4eae710f2a278246391d01b9ff9e1e87987e72f367c99612a7779171843bb74437860a843869648f5967af9c06767f669cf9a80f8f57f7eff8692cd04b19fba5d3fd25f030bba3acf1417c5aef0a355d95297e91c9d18ea9e03b009144b235079487557e393e1a261c75c06dc1a92ae46a4fcbc15834766091fbde00143f0f8a884bb541a17ec644e92452a79ee96c31aed01396d2cd843ba91448d55570c6c1ea16e7296a29443d003c5daed6d51c2d58e4b89c791606fac7009c23773f0eda7a90ea68e615c136e4d75da11d5f28364038830a6cb2f16935217de02ad0817de2c887c3fb23ca0d33abd333d50ba164c96adc5875177bec945306218f4dccd57700244bf0b604ca2c4de5c9f9df8fbfe6faaa07f970c85881), signing_key: 7cede71d0c20bcc3014635c79b4ca0d33759a3340f5799cba9368eb9ddc58df7, slots: 231..284 }, Validator { address: Address("95496bf079f8b824e61e52e4eecd4426d6b8cde1"), voting_key: LazyPublicKey(c8512dc4a69729170fe2ac5cf6c9b25322fa974ceee569b5d4f86a5238a51de896deb5e6998d59f90944aad3e47e676433d13137b1bb0cffeeb158ce5594c937294a8330475b79a2463fa752b1fc851e42383175a7f05b0183475da14129013e5b437436f6705ef9e4509391663806ff62e7abf8a49e583c3a5741e2272f014f50f686f2181849f05d43d82ed5d62f69a427529b4f981ce582ad811565db5e1b18c6b7a6bf6664c6d54eaf7fd7424fd6a92f13543568681a4115b15204003f3efe58109efd1af4c591bb3984aac1e74eee4ddd5f5f4900d62feb2ac1e046ccf52c66d8526c69a6ca72823dcf3e060d7e488f7aded41d6f94bc9876a4083d243cff5fa6e4758674270d18ec9c32fbd553049bd0394da82cfa7a69390600), signing_key: 81def5effc93ae283b5acfa466d0a544218b1bf3c3558f6cf6403bd9aed14dee, slots: 284..369 }, Validator { address: Address("a86db9670a7c1a28f8377d850304379594e99e36"), voting_key: LazyPublicKey(53a4acb81010e714c8b3e85888a39bbb8252ea7edea3dee1b7962cf5c57f6bf1cd110d9b150b9531cf0eac428bbdeb34fec6d61a837dbcdec6624630341326ee04a58058f26e4f2df83e6732530729be189c5502d2f911a87eb738cd5bb2012e8990c98cc12966a056f2471109f134d13c62a4da99ba91de0bf15d42b0b5c03fe942e1f8107d371bdae866ed1f9af6ea76f0ed39605f53b8c3058b682718640f1cf16ecaf330177567bc75e07eec37e4fb8d4eca45feeba36a92f98d34006b54c5c8c3ca926b071a0910035983ec5d144ae03d6771b2dc25528e23615b046d63cf06241d79c4c081d1ae79346ca84062c9f84420f5ee918fdb3d511e29011d65be59bfc260073c328fe03746d7c5ee19498a23e5cea84705e91995fb00), signing_key: b5487c57e8f22f18968dd68c51671649a80b496266b3239353cd767c606692de, slots: 369..434 }, Validator { address: Address("ded24a92947a31ae4495f5f134dc82c5a29f76df"), voting_key: LazyPublicKey(e6459600e07c11f28eb225fb9b2ca3ae8aaab06835afc557087e39f6177a505482f00d908dc401d342e84cd83327fbaff578d9bb6264b39b5a52905c0cb37d6f255eb49c1cf66e7828a94e44c1d7433154e9cd7b646b16224ee79cfdaaed00424d176afe578b7c2643dd03e1fd237129db30f772368911847592781b4d382413a6800997ee71f16daca8fd04dc689ae77922cea72b8c64fa4bfd2d08fcd182fcf81297b0b3d760beeb1a49bef91375ddc6af74413c596b8f9fd8634c7300477b7fa471a73fbf2c4a4c30fbc207f927b3cae8212f3f2a46c382de933e84aa7810648243843bdc81a2ad807917533eda1111ba38a18f903097941ed59733f8d3f5e364fa8bc379285e703a9af636b0a17618d99a68413225941bd622f780), signing_key: fc2d2b8e3b7abfa39b79d2c2969c55a9996dbf79f6bd2fe80fad7c810ac279ea, slots: 434..435 }, Validator { address: Address("e00787fb0be4e48212660aeb205a8a76b78d0392"), voting_key: LazyPublicKey(db0a7ab07398f84802819ea9d2c365c39b115962b2e3c92b83c2b76fb9bfccc9e326b818035853bf2c6219c1be5bbedd730a1d5346341b28e90baba646cb7f8ea9f398edb959b0f8891839361bedd46c4439ce125c65b821d1970cfdd38e00ff098bc6ae624fe7d175f2305af9189e4ca3293cf1d280fdf78cde13c375ee3fd5c1e840c87a9b5230cc655d5ba5968f96401dcd057561a890ad910149fc935249706d9095bdf66b884682ed44d008e0e645dfd76fa2d29a513e58d3c16900851a702131bcf6710a852b1e60f05913f3537e49eeffa2584fad057b15abd01f65dd1e06fd996c6edcb43af6b9e0a9feafdf5dc2c4a636da2f9f505249ddccef17fb03975c125766ac72b3eb80d3f686444978b70efd24be8e33c70bdfa980), signing_key: 38a15d65eaef2462702332026dbd18d3f2e3c692e02427f3033cfd6f126762c3, slots: 435..457 }, Validator { address: Address("e346ce21e99068990c5df271ed0ed25e094d3156"), voting_key: LazyPublicKey(e9859318d2985998a8753ced371682cd3fc3638ab0d75028d39305625b844ea452bd00e39cfdec4bf48d8cbc2b1fade2eb1c555f30270a4d391dff9dc55985a09137bca7e15248f8f446af49ddd16562ca3c95232386d901a7ae51de1dc0010b8d9785b198c8781083d4b09aaf5d58ba24d9c2bd29d88fc54cfbd8053c41bb2041f7ef0328ff32906b77ab1664e707c4da4723acce5ec2c2df39f4698d4cdd77b82b3a282b0ebf0034bc0f4fc3d7c11fd00cc8533ecabbdcb7dcf27d0e019775ca127da296a62a430c417bc8fbd8fe2c1d0bee0e291b6adab795e4f802adb10c9d589c8b7a6b01810291e9fe2b1a226496cf768f0d35f135d03afafcf16e0e2a75702280299b481e2e0e5ad7e6cb58ced498d8cce76446882778568400), signing_key: 9c630945f2764818e8ea0924b8cea68a6680aa344edfd8103fd882392556ca03, slots: 457..512 }], validator_map: {Address("1dab7b1fc3483d8822eb223a3a84bf5a7a033081"): 0, Address("6c6c822f592ff5e7b07d5acb284ea282c54884ec"): 1, Address("7a493b7fa61f4ed323c67f94d25d95f5719bdc9c"): 2, Address("83cbb5be504fdde49fb2948476a7aa204933d205"): 3, Address("95496bf079f8b824e61e52e4eecd4426d6b8cde1"): 4, Address("a86db9670a7c1a28f8377d850304379594e99e36"): 5, Address("ded24a92947a31ae4495f5f134dc82c5a29f76df"): 6, Address("e00787fb0be4e48212660aeb205a8a76b78d0392"): 7, Address("e346ce21e99068990c5df271ed0ed25e094d3156"): 8} }, locators: [5711919193f2b22822c42a990200ad0adaba43737a5e378531cf56300339451f, 6b1e5d0e649d17e7112279614277c8dcb07aafb56807ee08fac9d4cb9733ec01, ea5dd050b281931bb74bd2f544fd30f667d52df4a00be6e0e52a3654c4955806], include_micro_bodies: false, direction: Forward }
2024-03-04T08:28:34.210986000Z DEBUG sync_queue           | Re-requesting MissingBlockRequest { target_block_number: 17778523, target_block_hash: a7cd50e34dcf087199386aac6a26a8753af7df72003ce1fa9c2f028a3651e314, epoch_validators: Validators { validators: [Validator { address: Address("1dab7b1fc3483d8822eb223a3a84bf5a7a033081"), voting_key: LazyPublicKey(7c127cda41181fc2fff5ef49b38d5b4b791aa02bac26bbd227f8564a26d38eb2619901b8bb465c00f2fe68b15c3cf26e59dbaea8cb07882065827950442c052ceeb171ae3e454acbc347aca42fc71d14ced130218e7761169bc5a9fd740501ca774db39197e2bc4df5853c4e354fc3f4804e9a96b1cc33e2720b314097ccb553439b6b33446d10b7f76b1c7ea90882d56fd30c58d9454589ce1dea6b9e8295c9de1c8c0e5c6615a8c1c188948f0d5d4dffbc842ff97b50b2fc2a77f07000b3a58025fddca83aca027166c92469f3b84c88cbc894e7fed33bc70b1bedfaa47e1050196a228fde5e1dbec94e48a907b15a72d5f84ef6afc098e29fcd626a9d6eaff12eee1774d0f420b7dc5be943b8cd4afea715da95b90f63343e217300), signing_key: d419c718543cbeb6068e322800d53aeb7bc90f221fcc92c80247ccd914ced438, slots: 0..65 }, Validator { address: Address("6c6c822f592ff5e7b07d5acb284ea282c54884ec"), voting_key: LazyPublicKey(81f8c6e4d629b91fae9afb4fcb4ce5639002cc5a27b10669752cae00d60879133681dc113e4ea5acc05e1a157b2bf07ddcbfa449a8999f9420cc4fc57bd25fdb23d564a5d9de0337853415342090c17d5131fad9c1c81e8632410b803d3b00baf9bd234d3510512665f083a644eca7bbe41fc1c06c1cead3d4e4f52550182af411e0cff067bb749d23e3d22784be671902813062f04f7877c682dd6039f954caea0038704e91614b0389b6ae584e81526f16040297dceaf4ba994af5d100543d87de2dc0c033449d706b6a4dad44ea90f49e7fcbd888dec4e95cc35efaf04108335ef74f97730dd0c809b25a737f7f9e821800e27b2e3078b65aa4af516adc649a514f6cb46981f3bf080f312a872e3a120b709905310ff3b75fe06681), signing_key: b1881f7b5412d456a9acb7d9021d3b032686608c407b487ff377d07cd9fbf244, slots: 65..168 }, Validator { address: Address("7a493b7fa61f4ed323c67f94d25d95f5719bdc9c"), voting_key: LazyPublicKey(fcdd7c2a13dfa03fb29ea92c8edbd4315055d84114f75a3bc36fc311e6eae698e9cc5d6c7e1e4a4a0f937a694fa666e47eb6f1a21d44a4f9b69f2acf6f25e3289d77c3bb3caab154dee0e263ec79c0a263e5c84c45c7e2d84c8f7f96f356015be91276a5d6e4923009ccc42fff3474b51609f31dff4410b25017a265dc350c698a06597391fb67a15003df808da1cdf70aa313c80cae0e793c6289d7e420ab4084acf32d531ebf2eafbe73e6b2ab923f3589efc75ba8e9bb0c62a7e11e00116e3f1a4164540b01073decb05ca56c83ec6eacb4af0234910b1b0133f06556003601e5e00d25d905c68c9fb95e3c28859cadd49b23a2a1b5606d63de7b2f1b8806badef3fb188d9be4cb9215926762b6cf0743d44c8790544f0a77be2701), signing_key: ee1424f128e4b8ae9a4951b7fb4717fa94cd9c74f29dbffdf591cdedbe149a8d, slots: 168..231 }, Validator { address: Address("83cbb5be504fdde49fb2948476a7aa204933d205"), voting_key: LazyPublicKey(23dede6c4eae710f2a278246391d01b9ff9e1e87987e72f367c99612a7779171843bb74437860a843869648f5967af9c06767f669cf9a80f8f57f7eff8692cd04b19fba5d3fd25f030bba3acf1417c5aef0a355d95297e91c9d18ea9e03b009144b235079487557e393e1a261c75c06dc1a92ae46a4fcbc15834766091fbde00143f0f8a884bb541a17ec644e92452a79ee96c31aed01396d2cd843ba91448d55570c6c1ea16e7296a29443d003c5daed6d51c2d58e4b89c791606fac7009c23773f0eda7a90ea68e615c136e4d75da11d5f28364038830a6cb2f16935217de02ad0817de2c887c3fb23ca0d33abd333d50ba164c96adc5875177bec945306218f4dccd57700244bf0b604ca2c4de5c9f9df8fbfe6faaa07f970c85881), signing_key: 7cede71d0c20bcc3014635c79b4ca0d33759a3340f5799cba9368eb9ddc58df7, slots: 231..284 }, Validator { address: Address("95496bf079f8b824e61e52e4eecd4426d6b8cde1"), voting_key: LazyPublicKey(c8512dc4a69729170fe2ac5cf6c9b25322fa974ceee569b5d4f86a5238a51de896deb5e6998d59f90944aad3e47e676433d13137b1bb0cffeeb158ce5594c937294a8330475b79a2463fa752b1fc851e42383175a7f05b0183475da14129013e5b437436f6705ef9e4509391663806ff62e7abf8a49e583c3a5741e2272f014f50f686f2181849f05d43d82ed5d62f69a427529b4f981ce582ad811565db5e1b18c6b7a6bf6664c6d54eaf7fd7424fd6a92f13543568681a4115b15204003f3efe58109efd1af4c591bb3984aac1e74eee4ddd5f5f4900d62feb2ac1e046ccf52c66d8526c69a6ca72823dcf3e060d7e488f7aded41d6f94bc9876a4083d243cff5fa6e4758674270d18ec9c32fbd553049bd0394da82cfa7a69390600), signing_key: 81def5effc93ae283b5acfa466d0a544218b1bf3c3558f6cf6403bd9aed14dee, slots: 284..369 }, Validator { address: Address("a86db9670a7c1a28f8377d850304379594e99e36"), voting_key: LazyPublicKey(53a4acb81010e714c8b3e85888a39bbb8252ea7edea3dee1b7962cf5c57f6bf1cd110d9b150b9531cf0eac428bbdeb34fec6d61a837dbcdec6624630341326ee04a58058f26e4f2df83e6732530729be189c5502d2f911a87eb738cd5bb2012e8990c98cc12966a056f2471109f134d13c62a4da99ba91de0bf15d42b0b5c03fe942e1f8107d371bdae866ed1f9af6ea76f0ed39605f53b8c3058b682718640f1cf16ecaf330177567bc75e07eec37e4fb8d4eca45feeba36a92f98d34006b54c5c8c3ca926b071a0910035983ec5d144ae03d6771b2dc25528e23615b046d63cf06241d79c4c081d1ae79346ca84062c9f84420f5ee918fdb3d511e29011d65be59bfc260073c328fe03746d7c5ee19498a23e5cea84705e91995fb00), signing_key: b5487c57e8f22f18968dd68c51671649a80b496266b3239353cd767c606692de, slots: 369..434 }, Validator { address: Address("ded24a92947a31ae4495f5f134dc82c5a29f76df"), voting_key: LazyPublicKey(e6459600e07c11f28eb225fb9b2ca3ae8aaab06835afc557087e39f6177a505482f00d908dc401d342e84cd83327fbaff578d9bb6264b39b5a52905c0cb37d6f255eb49c1cf66e7828a94e44c1d7433154e9cd7b646b16224ee79cfdaaed00424d176afe578b7c2643dd03e1fd237129db30f772368911847592781b4d382413a6800997ee71f16daca8fd04dc689ae77922cea72b8c64fa4bfd2d08fcd182fcf81297b0b3d760beeb1a49bef91375ddc6af74413c596b8f9fd8634c7300477b7fa471a73fbf2c4a4c30fbc207f927b3cae8212f3f2a46c382de933e84aa7810648243843bdc81a2ad807917533eda1111ba38a18f903097941ed59733f8d3f5e364fa8bc379285e703a9af636b0a17618d99a68413225941bd622f780), signing_key: fc2d2b8e3b7abfa39b79d2c2969c55a9996dbf79f6bd2fe80fad7c810ac279ea, slots: 434..435 }, Validator { address: Address("e00787fb0be4e48212660aeb205a8a76b78d0392"), voting_key: LazyPublicKey(db0a7ab07398f84802819ea9d2c365c39b115962b2e3c92b83c2b76fb9bfccc9e326b818035853bf2c6219c1be5bbedd730a1d5346341b28e90baba646cb7f8ea9f398edb959b0f8891839361bedd46c4439ce125c65b821d1970cfdd38e00ff098bc6ae624fe7d175f2305af9189e4ca3293cf1d280fdf78cde13c375ee3fd5c1e840c87a9b5230cc655d5ba5968f96401dcd057561a890ad910149fc935249706d9095bdf66b884682ed44d008e0e645dfd76fa2d29a513e58d3c16900851a702131bcf6710a852b1e60f05913f3537e49eeffa2584fad057b15abd01f65dd1e06fd996c6edcb43af6b9e0a9feafdf5dc2c4a636da2f9f505249ddccef17fb03975c125766ac72b3eb80d3f686444978b70efd24be8e33c70bdfa980), signing_key: 38a15d65eaef2462702332026dbd18d3f2e3c692e02427f3033cfd6f126762c3, slots: 435..457 }, Validator { address: Address("e346ce21e99068990c5df271ed0ed25e094d3156"), voting_key: LazyPublicKey(e9859318d2985998a8753ced371682cd3fc3638ab0d75028d39305625b844ea452bd00e39cfdec4bf48d8cbc2b1fade2eb1c555f30270a4d391dff9dc55985a09137bca7e15248f8f446af49ddd16562ca3c95232386d901a7ae51de1dc0010b8d9785b198c8781083d4b09aaf5d58ba24d9c2bd29d88fc54cfbd8053c41bb2041f7ef0328ff32906b77ab1664e707c4da4723acce5ec2c2df39f4698d4cdd77b82b3a282b0ebf0034bc0f4fc3d7c11fd00cc8533ecabbdcb7dcf27d0e019775ca127da296a62a430c417bc8fbd8fe2c1d0bee0e291b6adab795e4f802adb10c9d589c8b7a6b01810291e9fe2b1a226496cf768f0d35f135d03afafcf16e0e2a75702280299b481e2e0e5ad7e6cb58ced498d8cce76446882778568400), signing_key: 9c630945f2764818e8ea0924b8cea68a6680aa344edfd8103fd882392556ca03, slots: 457..512 }], validator_map: {Address("1dab7b1fc3483d8822eb223a3a84bf5a7a033081"): 0, Address("6c6c822f592ff5e7b07d5acb284ea282c54884ec"): 1, Address("7a493b7fa61f4ed323c67f94d25d95f5719bdc9c"): 2, Address("83cbb5be504fdde49fb2948476a7aa204933d205"): 3, Address("95496bf079f8b824e61e52e4eecd4426d6b8cde1"): 4, Address("a86db9670a7c1a28f8377d850304379594e99e36"): 5, Address("ded24a92947a31ae4495f5f134dc82c5a29f76df"): 6, Address("e00787fb0be4e48212660aeb205a8a76b78d0392"): 7, Address("e346ce21e99068990c5df271ed0ed25e094d3156"): 8} }, locators: [5711919193f2b22822c42a990200ad0adaba43737a5e378531cf56300339451f, 6b1e5d0e649d17e7112279614277c8dcb07aafb56807ee08fac9d4cb9733ec01, ea5dd050b281931bb74bd2f544fd30f667d52df4a00be6e0e52a3654c4955806], include_micro_bodies: false, direction: Forward } @ 129 peer_id=12D3KooWHjxZ465uHKw3JE8vKdrJxkZ1GMgKAmsUnmBeQGa28nys current_peer_index=2
2024-03-04T08:28:34.296756000Z INFO  nimiq_client         | Consensus: established - Head: #17778526:MI:5711919193 - Peers: 6 consensus_established=true block_number=17778526 num_peers=6
2024-03-04T08:28:34.428581000Z DEBUG block_request_compo… | Received empty missing blocks response
2024-03-04T08:28:34.428654000Z DEBUG sync_queue           | Verification failed peer_id=3 id=MissingBlockRequest { target_block_number: 17778523, target_block_hash: a7cd50e34dcf087199386aac6a26a8753af7df72003ce1fa9c2f028a3651e314, epoch_validators: Validators { validators: [Validator { address: Address("1dab7b1fc3483d8822eb223a3a84bf5a7a033081"), voting_key: LazyPublicKey(7c127cda41181fc2fff5ef49b38d5b4b791aa02bac26bbd227f8564a26d38eb2619901b8bb465c00f2fe68b15c3cf26e59dbaea8cb07882065827950442c052ceeb171ae3e454acbc347aca42fc71d14ced130218e7761169bc5a9fd740501ca774db39197e2bc4df5853c4e354fc3f4804e9a96b1cc33e2720b314097ccb553439b6b33446d10b7f76b1c7ea90882d56fd30c58d9454589ce1dea6b9e8295c9de1c8c0e5c6615a8c1c188948f0d5d4dffbc842ff97b50b2fc2a77f07000b3a58025fddca83aca027166c92469f3b84c88cbc894e7fed33bc70b1bedfaa47e1050196a228fde5e1dbec94e48a907b15a72d5f84ef6afc098e29fcd626a9d6eaff12eee1774d0f420b7dc5be943b8cd4afea715da95b90f63343e217300), signing_key: d419c718543cbeb6068e322800d53aeb7bc90f221fcc92c80247ccd914ced438, slots: 0..65 }, Validator { address: Address("6c6c822f592ff5e7b07d5acb284ea282c54884ec"), voting_key: LazyPublicKey(81f8c6e4d629b91fae9afb4fcb4ce5639002cc5a27b10669752cae00d60879133681dc113e4ea5acc05e1a157b2bf07ddcbfa449a8999f9420cc4fc57bd25fdb23d564a5d9de0337853415342090c17d5131fad9c1c81e8632410b803d3b00baf9bd234d3510512665f083a644eca7bbe41fc1c06c1cead3d4e4f52550182af411e0cff067bb749d23e3d22784be671902813062f04f7877c682dd6039f954caea0038704e91614b0389b6ae584e81526f16040297dceaf4ba994af5d100543d87de2dc0c033449d706b6a4dad44ea90f49e7fcbd888dec4e95cc35efaf04108335ef74f97730dd0c809b25a737f7f9e821800e27b2e3078b65aa4af516adc649a514f6cb46981f3bf080f312a872e3a120b709905310ff3b75fe06681), signing_key: b1881f7b5412d456a9acb7d9021d3b032686608c407b487ff377d07cd9fbf244, slots: 65..168 }, Validator { address: Address("7a493b7fa61f4ed323c67f94d25d95f5719bdc9c"), voting_key: LazyPublicKey(fcdd7c2a13dfa03fb29ea92c8edbd4315055d84114f75a3bc36fc311e6eae698e9cc5d6c7e1e4a4a0f937a694fa666e47eb6f1a21d44a4f9b69f2acf6f25e3289d77c3bb3caab154dee0e263ec79c0a263e5c84c45c7e2d84c8f7f96f356015be91276a5d6e4923009ccc42fff3474b51609f31dff4410b25017a265dc350c698a06597391fb67a15003df808da1cdf70aa313c80cae0e793c6289d7e420ab4084acf32d531ebf2eafbe73e6b2ab923f3589efc75ba8e9bb0c62a7e11e00116e3f1a4164540b01073decb05ca56c83ec6eacb4af0234910b1b0133f06556003601e5e00d25d905c68c9fb95e3c28859cadd49b23a2a1b5606d63de7b2f1b8806badef3fb188d9be4cb9215926762b6cf0743d44c8790544f0a77be2701), signing_key: ee1424f128e4b8ae9a4951b7fb4717fa94cd9c74f29dbffdf591cdedbe149a8d, slots: 168..231 }, Validator { address: Address("83cbb5be504fdde49fb2948476a7aa204933d205"), voting_key: LazyPublicKey(23dede6c4eae710f2a278246391d01b9ff9e1e87987e72f367c99612a7779171843bb74437860a843869648f5967af9c06767f669cf9a80f8f57f7eff8692cd04b19fba5d3fd25f030bba3acf1417c5aef0a355d95297e91c9d18ea9e03b009144b235079487557e393e1a261c75c06dc1a92ae46a4fcbc15834766091fbde00143f0f8a884bb541a17ec644e92452a79ee96c31aed01396d2cd843ba91448d55570c6c1ea16e7296a29443d003c5daed6d51c2d58e4b89c791606fac7009c23773f0eda7a90ea68e615c136e4d75da11d5f28364038830a6cb2f16935217de02ad0817de2c887c3fb23ca0d33abd333d50ba164c96adc5875177bec945306218f4dccd57700244bf0b604ca2c4de5c9f9df8fbfe6faaa07f970c85881), signing_key: 7cede71d0c20bcc3014635c79b4ca0d33759a3340f5799cba9368eb9ddc58df7, slots: 231..284 }, Validator { address: Address("95496bf079f8b824e61e52e4eecd4426d6b8cde1"), voting_key: LazyPublicKey(c8512dc4a69729170fe2ac5cf6c9b25322fa974ceee569b5d4f86a5238a51de896deb5e6998d59f90944aad3e47e676433d13137b1bb0cffeeb158ce5594c937294a8330475b79a2463fa752b1fc851e42383175a7f05b0183475da14129013e5b437436f6705ef9e4509391663806ff62e7abf8a49e583c3a5741e2272f014f50f686f2181849f05d43d82ed5d62f69a427529b4f981ce582ad811565db5e1b18c6b7a6bf6664c6d54eaf7fd7424fd6a92f13543568681a4115b15204003f3efe58109efd1af4c591bb3984aac1e74eee4ddd5f5f4900d62feb2ac1e046ccf52c66d8526c69a6ca72823dcf3e060d7e488f7aded41d6f94bc9876a4083d243cff5fa6e4758674270d18ec9c32fbd553049bd0394da82cfa7a69390600), signing_key: 81def5effc93ae283b5acfa466d0a544218b1bf3c3558f6cf6403bd9aed14dee, slots: 284..369 }, Validator { address: Address("a86db9670a7c1a28f8377d850304379594e99e36"), voting_key: LazyPublicKey(53a4acb81010e714c8b3e85888a39bbb8252ea7edea3dee1b7962cf5c57f6bf1cd110d9b150b9531cf0eac428bbdeb34fec6d61a837dbcdec6624630341326ee04a58058f26e4f2df83e6732530729be189c5502d2f911a87eb738cd5bb2012e8990c98cc12966a056f2471109f134d13c62a4da99ba91de0bf15d42b0b5c03fe942e1f8107d371bdae866ed1f9af6ea76f0ed39605f53b8c3058b682718640f1cf16ecaf330177567bc75e07eec37e4fb8d4eca45feeba36a92f98d34006b54c5c8c3ca926b071a0910035983ec5d144ae03d6771b2dc25528e23615b046d63cf06241d79c4c081d1ae79346ca84062c9f84420f5ee918fdb3d511e29011d65be59bfc260073c328fe03746d7c5ee19498a23e5cea84705e91995fb00), signing_key: b5487c57e8f22f18968dd68c51671649a80b496266b3239353cd767c606692de, slots: 369..434 }, Validator { address: Address("ded24a92947a31ae4495f5f134dc82c5a29f76df"), voting_key: LazyPublicKey(e6459600e07c11f28eb225fb9b2ca3ae8aaab06835afc557087e39f6177a505482f00d908dc401d342e84cd83327fbaff578d9bb6264b39b5a52905c0cb37d6f255eb49c1cf66e7828a94e44c1d7433154e9cd7b646b16224ee79cfdaaed00424d176afe578b7c2643dd03e1fd237129db30f772368911847592781b4d382413a6800997ee71f16daca8fd04dc689ae77922cea72b8c64fa4bfd2d08fcd182fcf81297b0b3d760beeb1a49bef91375ddc6af74413c596b8f9fd8634c7300477b7fa471a73fbf2c4a4c30fbc207f927b3cae8212f3f2a46c382de933e84aa7810648243843bdc81a2ad807917533eda1111ba38a18f903097941ed59733f8d3f5e364fa8bc379285e703a9af636b0a17618d99a68413225941bd622f780), signing_key: fc2d2b8e3b7abfa39b79d2c2969c55a9996dbf79f6bd2fe80fad7c810ac279ea, slots: 434..435 }, Validator { address: Address("e00787fb0be4e48212660aeb205a8a76b78d0392"), voting_key: LazyPublicKey(db0a7ab07398f84802819ea9d2c365c39b115962b2e3c92b83c2b76fb9bfccc9e326b818035853bf2c6219c1be5bbedd730a1d5346341b28e90baba646cb7f8ea9f398edb959b0f8891839361bedd46c4439ce125c65b821d1970cfdd38e00ff098bc6ae624fe7d175f2305af9189e4ca3293cf1d280fdf78cde13c375ee3fd5c1e840c87a9b5230cc655d5ba5968f96401dcd057561a890ad910149fc935249706d9095bdf66b884682ed44d008e0e645dfd76fa2d29a513e58d3c16900851a702131bcf6710a852b1e60f05913f3537e49eeffa2584fad057b15abd01f65dd1e06fd996c6edcb43af6b9e0a9feafdf5dc2c4a636da2f9f505249ddccef17fb03975c125766ac72b3eb80d3f686444978b70efd24be8e33c70bdfa980), signing_key: 38a15d65eaef2462702332026dbd18d3f2e3c692e02427f3033cfd6f126762c3, slots: 435..457 }, Validator { address: Address("e346ce21e99068990c5df271ed0ed25e094d3156"), voting_key: LazyPublicKey(e9859318d2985998a8753ced371682cd3fc3638ab0d75028d39305625b844ea452bd00e39cfdec4bf48d8cbc2b1fade2eb1c555f30270a4d391dff9dc55985a09137bca7e15248f8f446af49ddd16562ca3c95232386d901a7ae51de1dc0010b8d9785b198c8781083d4b09aaf5d58ba24d9c2bd29d88fc54cfbd8053c41bb2041f7ef0328ff32906b77ab1664e707c4da4723acce5ec2c2df39f4698d4cdd77b82b3a282b0ebf0034bc0f4fc3d7c11fd00cc8533ecabbdcb7dcf27d0e019775ca127da296a62a430c417bc8fbd8fe2c1d0bee0e291b6adab795e4f802adb10c9d589c8b7a6b01810291e9fe2b1a226496cf768f0d35f135d03afafcf16e0e2a75702280299b481e2e0e5ad7e6cb58ced498d8cce76446882778568400), signing_key: 9c630945f2764818e8ea0924b8cea68a6680aa344edfd8103fd882392556ca03, slots: 457..512 }], validator_map: {Address("1dab7b1fc3483d8822eb223a3a84bf5a7a033081"): 0, Address("6c6c822f592ff5e7b07d5acb284ea282c54884ec"): 1, Address("7a493b7fa61f4ed323c67f94d25d95f5719bdc9c"): 2, Address("83cbb5be504fdde49fb2948476a7aa204933d205"): 3, Address("95496bf079f8b824e61e52e4eecd4426d6b8cde1"): 4, Address("a86db9670a7c1a28f8377d850304379594e99e36"): 5, Address("ded24a92947a31ae4495f5f134dc82c5a29f76df"): 6, Address("e00787fb0be4e48212660aeb205a8a76b78d0392"): 7, Address("e346ce21e99068990c5df271ed0ed25e094d3156"): 8} }, locators: [5711919193f2b22822c42a990200ad0adaba43737a5e378531cf56300339451f, 6b1e5d0e649d17e7112279614277c8dcb07aafb56807ee08fac9d4cb9733ec01, ea5dd050b281931bb74bd2f544fd30f667d52df4a00be6e0e52a3654c4955806], include_micro_bodies: false, direction: Forward }
2024-03-04T08:28:34.430807000Z DEBUG sync_queue           | Re-requesting MissingBlockRequest { target_block_number: 17778523, target_block_hash: a7cd50e34dcf087199386aac6a26a8753af7df72003ce1fa9c2f028a3651e314, epoch_validators: Validators { validators: [Validator { address: Address("1dab7b1fc3483d8822eb223a3a84bf5a7a033081"), voting_key: LazyPublicKey(7c127cda41181fc2fff5ef49b38d5b4b791aa02bac26bbd227f8564a26d38eb2619901b8bb465c00f2fe68b15c3cf26e59dbaea8cb07882065827950442c052ceeb171ae3e454acbc347aca42fc71d14ced130218e7761169bc5a9fd740501ca774db39197e2bc4df5853c4e354fc3f4804e9a96b1cc33e2720b314097ccb553439b6b33446d10b7f76b1c7ea90882d56fd30c58d9454589ce1dea6b9e8295c9de1c8c0e5c6615a8c1c188948f0d5d4dffbc842ff97b50b2fc2a77f07000b3a58025fddca83aca027166c92469f3b84c88cbc894e7fed33bc70b1bedfaa47e1050196a228fde5e1dbec94e48a907b15a72d5f84ef6afc098e29fcd626a9d6eaff12eee1774d0f420b7dc5be943b8cd4afea715da95b90f63343e217300), signing_key: d419c718543cbeb6068e322800d53aeb7bc90f221fcc92c80247ccd914ced438, slots: 0..65 }, Validator { address: Address("6c6c822f592ff5e7b07d5acb284ea282c54884ec"), voting_key: LazyPublicKey(81f8c6e4d629b91fae9afb4fcb4ce5639002cc5a27b10669752cae00d60879133681dc113e4ea5acc05e1a157b2bf07ddcbfa449a8999f9420cc4fc57bd25fdb23d564a5d9de0337853415342090c17d5131fad9c1c81e8632410b803d3b00baf9bd234d3510512665f083a644eca7bbe41fc1c06c1cead3d4e4f52550182af411e0cff067bb749d23e3d22784be671902813062f04f7877c682dd6039f954caea0038704e91614b0389b6ae584e81526f16040297dceaf4ba994af5d100543d87de2dc0c033449d706b6a4dad44ea90f49e7fcbd888dec4e95cc35efaf04108335ef74f97730dd0c809b25a737f7f9e821800e27b2e3078b65aa4af516adc649a514f6cb46981f3bf080f312a872e3a120b709905310ff3b75fe06681), signing_key: b1881f7b5412d456a9acb7d9021d3b032686608c407b487ff377d07cd9fbf244, slots: 65..168 }, Validator { address: Address("7a493b7fa61f4ed323c67f94d25d95f5719bdc9c"), voting_key: LazyPublicKey(fcdd7c2a13dfa03fb29ea92c8edbd4315055d84114f75a3bc36fc311e6eae698e9cc5d6c7e1e4a4a0f937a694fa666e47eb6f1a21d44a4f9b69f2acf6f25e3289d77c3bb3caab154dee0e263ec79c0a263e5c84c45c7e2d84c8f7f96f356015be91276a5d6e4923009ccc42fff3474b51609f31dff4410b25017a265dc350c698a06597391fb67a15003df808da1cdf70aa313c80cae0e793c6289d7e420ab4084acf32d531ebf2eafbe73e6b2ab923f3589efc75ba8e9bb0c62a7e11e00116e3f1a4164540b01073decb05ca56c83ec6eacb4af0234910b1b0133f06556003601e5e00d25d905c68c9fb95e3c28859cadd49b23a2a1b5606d63de7b2f1b8806badef3fb188d9be4cb9215926762b6cf0743d44c8790544f0a77be2701), signing_key: ee1424f128e4b8ae9a4951b7fb4717fa94cd9c74f29dbffdf591cdedbe149a8d, slots: 168..231 }, Validator { address: Address("83cbb5be504fdde49fb2948476a7aa204933d205"), voting_key: LazyPublicKey(23dede6c4eae710f2a278246391d01b9ff9e1e87987e72f367c99612a7779171843bb74437860a843869648f5967af9c06767f669cf9a80f8f57f7eff8692cd04b19fba5d3fd25f030bba3acf1417c5aef0a355d95297e91c9d18ea9e03b009144b235079487557e393e1a261c75c06dc1a92ae46a4fcbc15834766091fbde00143f0f8a884bb541a17ec644e92452a79ee96c31aed01396d2cd843ba91448d55570c6c1ea16e7296a29443d003c5daed6d51c2d58e4b89c791606fac7009c23773f0eda7a90ea68e615c136e4d75da11d5f28364038830a6cb2f16935217de02ad0817de2c887c3fb23ca0d33abd333d50ba164c96adc5875177bec945306218f4dccd57700244bf0b604ca2c4de5c9f9df8fbfe6faaa07f970c85881), signing_key: 7cede71d0c20bcc3014635c79b4ca0d33759a3340f5799cba9368eb9ddc58df7, slots: 231..284 }, Validator { address: Address("95496bf079f8b824e61e52e4eecd4426d6b8cde1"), voting_key: LazyPublicKey(c8512dc4a69729170fe2ac5cf6c9b25322fa974ceee569b5d4f86a5238a51de896deb5e6998d59f90944aad3e47e676433d13137b1bb0cffeeb158ce5594c937294a8330475b79a2463fa752b1fc851e42383175a7f05b0183475da14129013e5b437436f6705ef9e4509391663806ff62e7abf8a49e583c3a5741e2272f014f50f686f2181849f05d43d82ed5d62f69a427529b4f981ce582ad811565db5e1b18c6b7a6bf6664c6d54eaf7fd7424fd6a92f13543568681a4115b15204003f3efe58109efd1af4c591bb3984aac1e74eee4ddd5f5f4900d62feb2ac1e046ccf52c66d8526c69a6ca72823dcf3e060d7e488f7aded41d6f94bc9876a4083d243cff5fa6e4758674270d18ec9c32fbd553049bd0394da82cfa7a69390600), signing_key: 81def5effc93ae283b5acfa466d0a544218b1bf3c3558f6cf6403bd9aed14dee, slots: 284..369 }, Validator { address: Address("a86db9670a7c1a28f8377d850304379594e99e36"), voting_key: LazyPublicKey(53a4acb81010e714c8b3e85888a39bbb8252ea7edea3dee1b7962cf5c57f6bf1cd110d9b150b9531cf0eac428bbdeb34fec6d61a837dbcdec6624630341326ee04a58058f26e4f2df83e6732530729be189c5502d2f911a87eb738cd5bb2012e8990c98cc12966a056f2471109f134d13c62a4da99ba91de0bf15d42b0b5c03fe942e1f8107d371bdae866ed1f9af6ea76f0ed39605f53b8c3058b682718640f1cf16ecaf330177567bc75e07eec37e4fb8d4eca45feeba36a92f98d34006b54c5c8c3ca926b071a0910035983ec5d144ae03d6771b2dc25528e23615b046d63cf06241d79c4c081d1ae79346ca84062c9f84420f5ee918fdb3d511e29011d65be59bfc260073c328fe03746d7c5ee19498a23e5cea84705e91995fb00), signing_key: b5487c57e8f22f18968dd68c51671649a80b496266b3239353cd767c606692de, slots: 369..434 }, Validator { address: Address("ded24a92947a31ae4495f5f134dc82c5a29f76df"), voting_key: LazyPublicKey(e6459600e07c11f28eb225fb9b2ca3ae8aaab06835afc557087e39f6177a505482f00d908dc401d342e84cd83327fbaff578d9bb6264b39b5a52905c0cb37d6f255eb49c1cf66e7828a94e44c1d7433154e9cd7b646b16224ee79cfdaaed00424d176afe578b7c2643dd03e1fd237129db30f772368911847592781b4d382413a6800997ee71f16daca8fd04dc689ae77922cea72b8c64fa4bfd2d08fcd182fcf81297b0b3d760beeb1a49bef91375ddc6af74413c596b8f9fd8634c7300477b7fa471a73fbf2c4a4c30fbc207f927b3cae8212f3f2a46c382de933e84aa7810648243843bdc81a2ad807917533eda1111ba38a18f903097941ed59733f8d3f5e364fa8bc379285e703a9af636b0a17618d99a68413225941bd622f780), signing_key: fc2d2b8e3b7abfa39b79d2c2969c55a9996dbf79f6bd2fe80fad7c810ac279ea, slots: 434..435 }, Validator { address: Address("e00787fb0be4e48212660aeb205a8a76b78d0392"), voting_key: LazyPublicKey(db0a7ab07398f84802819ea9d2c365c39b115962b2e3c92b83c2b76fb9bfccc9e326b818035853bf2c6219c1be5bbedd730a1d5346341b28e90baba646cb7f8ea9f398edb959b0f8891839361bedd46c4439ce125c65b821d1970cfdd38e00ff098bc6ae624fe7d175f2305af9189e4ca3293cf1d280fdf78cde13c375ee3fd5c1e840c87a9b5230cc655d5ba5968f96401dcd057561a890ad910149fc935249706d9095bdf66b884682ed44d008e0e645dfd76fa2d29a513e58d3c16900851a702131bcf6710a852b1e60f05913f3537e49eeffa2584fad057b15abd01f65dd1e06fd996c6edcb43af6b9e0a9feafdf5dc2c4a636da2f9f505249ddccef17fb03975c125766ac72b3eb80d3f686444978b70efd24be8e33c70bdfa980), signing_key: 38a15d65eaef2462702332026dbd18d3f2e3c692e02427f3033cfd6f126762c3, slots: 435..457 }, Validator { address: Address("e346ce21e99068990c5df271ed0ed25e094d3156"), voting_key: LazyPublicKey(e9859318d2985998a8753ced371682cd3fc3638ab0d75028d39305625b844ea452bd00e39cfdec4bf48d8cbc2b1fade2eb1c555f30270a4d391dff9dc55985a09137bca7e15248f8f446af49ddd16562ca3c95232386d901a7ae51de1dc0010b8d9785b198c8781083d4b09aaf5d58ba24d9c2bd29d88fc54cfbd8053c41bb2041f7ef0328ff32906b77ab1664e707c4da4723acce5ec2c2df39f4698d4cdd77b82b3a282b0ebf0034bc0f4fc3d7c11fd00cc8533ecabbdcb7dcf27d0e019775ca127da296a62a430c417bc8fbd8fe2c1d0bee0e291b6adab795e4f802adb10c9d589c8b7a6b01810291e9fe2b1a226496cf768f0d35f135d03afafcf16e0e2a75702280299b481e2e0e5ad7e6cb58ced498d8cce76446882778568400), signing_key: 9c630945f2764818e8ea0924b8cea68a6680aa344edfd8103fd882392556ca03, slots: 457..512 }], validator_map: {Address("1dab7b1fc3483d8822eb223a3a84bf5a7a033081"): 0, Address("6c6c822f592ff5e7b07d5acb284ea282c54884ec"): 1, Address("7a493b7fa61f4ed323c67f94d25d95f5719bdc9c"): 2, Address("83cbb5be504fdde49fb2948476a7aa204933d205"): 3, Address("95496bf079f8b824e61e52e4eecd4426d6b8cde1"): 4, Address("a86db9670a7c1a28f8377d850304379594e99e36"): 5, Address("ded24a92947a31ae4495f5f134dc82c5a29f76df"): 6, Address("e00787fb0be4e48212660aeb205a8a76b78d0392"): 7, Address("e346ce21e99068990c5df271ed0ed25e094d3156"): 8} }, locators: [5711919193f2b22822c42a990200ad0adaba43737a5e378531cf56300339451f, 6b1e5d0e649d17e7112279614277c8dcb07aafb56807ee08fac9d4cb9733ec01, ea5dd050b281931bb74bd2f544fd30f667d52df4a00be6e0e52a3654c4955806], include_micro_bodies: false, direction: Forward } @ 129 peer_id=12D3KooWJkTbDJRFB8nAakcBLUWQKcZTZV1GM2xR141PiMb1wqg6 current_peer_index=2
2024-03-04T08:28:34.469993000Z DEBUG block_request_compo… | Received empty missing blocks response
2024-03-04T08:28:34.470079000Z DEBUG sync_queue           | Verification failed peer_id=0 id=MissingBlockRequest { target_block_number: 17778523, target_block_hash: a7cd50e34dcf087199386aac6a26a8753af7df72003ce1fa9c2f028a3651e314, epoch_validators: Validators { validators: [Validator { address: Address("1dab7b1fc3483d8822eb223a3a84bf5a7a033081"), voting_key: LazyPublicKey(7c127cda41181fc2fff5ef49b38d5b4b791aa02bac26bbd227f8564a26d38eb2619901b8bb465c00f2fe68b15c3cf26e59dbaea8cb07882065827950442c052ceeb171ae3e454acbc347aca42fc71d14ced130218e7761169bc5a9fd740501ca774db39197e2bc4df5853c4e354fc3f4804e9a96b1cc33e2720b314097ccb553439b6b33446d10b7f76b1c7ea90882d56fd30c58d9454589ce1dea6b9e8295c9de1c8c0e5c6615a8c1c188948f0d5d4dffbc842ff97b50b2fc2a77f07000b3a58025fddca83aca027166c92469f3b84c88cbc894e7fed33bc70b1bedfaa47e1050196a228fde5e1dbec94e48a907b15a72d5f84ef6afc098e29fcd626a9d6eaff12eee1774d0f420b7dc5be943b8cd4afea715da95b90f63343e217300), signing_key: d419c718543cbeb6068e322800d53aeb7bc90f221fcc92c80247ccd914ced438, slots: 0..65 }, Validator { address: Address("6c6c822f592ff5e7b07d5acb284ea282c54884ec"), voting_key: LazyPublicKey(81f8c6e4d629b91fae9afb4fcb4ce5639002cc5a27b10669752cae00d60879133681dc113e4ea5acc05e1a157b2bf07ddcbfa449a8999f9420cc4fc57bd25fdb23d564a5d9de0337853415342090c17d5131fad9c1c81e8632410b803d3b00baf9bd234d3510512665f083a644eca7bbe41fc1c06c1cead3d4e4f52550182af411e0cff067bb749d23e3d22784be671902813062f04f7877c682dd6039f954caea0038704e91614b0389b6ae584e81526f16040297dceaf4ba994af5d100543d87de2dc0c033449d706b6a4dad44ea90f49e7fcbd888dec4e95cc35efaf04108335ef74f97730dd0c809b25a737f7f9e821800e27b2e3078b65aa4af516adc649a514f6cb46981f3bf080f312a872e3a120b709905310ff3b75fe06681), signing_key: b1881f7b5412d456a9acb7d9021d3b032686608c407b487ff377d07cd9fbf244, slots: 65..168 }, Validator { address: Address("7a493b7fa61f4ed323c67f94d25d95f5719bdc9c"), voting_key: LazyPublicKey(fcdd7c2a13dfa03fb29ea92c8edbd4315055d84114f75a3bc36fc311e6eae698e9cc5d6c7e1e4a4a0f937a694fa666e47eb6f1a21d44a4f9b69f2acf6f25e3289d77c3bb3caab154dee0e263ec79c0a263e5c84c45c7e2d84c8f7f96f356015be91276a5d6e4923009ccc42fff3474b51609f31dff4410b25017a265dc350c698a06597391fb67a15003df808da1cdf70aa313c80cae0e793c6289d7e420ab4084acf32d531ebf2eafbe73e6b2ab923f3589efc75ba8e9bb0c62a7e11e00116e3f1a4164540b01073decb05ca56c83ec6eacb4af0234910b1b0133f06556003601e5e00d25d905c68c9fb95e3c28859cadd49b23a2a1b5606d63de7b2f1b8806badef3fb188d9be4cb9215926762b6cf0743d44c8790544f0a77be2701), signing_key: ee1424f128e4b8ae9a4951b7fb4717fa94cd9c74f29dbffdf591cdedbe149a8d, slots: 168..231 }, Validator { address: Address("83cbb5be504fdde49fb2948476a7aa204933d205"), voting_key: LazyPublicKey(23dede6c4eae710f2a278246391d01b9ff9e1e87987e72f367c99612a7779171843bb74437860a843869648f5967af9c06767f669cf9a80f8f57f7eff8692cd04b19fba5d3fd25f030bba3acf1417c5aef0a355d95297e91c9d18ea9e03b009144b235079487557e393e1a261c75c06dc1a92ae46a4fcbc15834766091fbde00143f0f8a884bb541a17ec644e92452a79ee96c31aed01396d2cd843ba91448d55570c6c1ea16e7296a29443d003c5daed6d51c2d58e4b89c791606fac7009c23773f0eda7a90ea68e615c136e4d75da11d5f28364038830a6cb2f16935217de02ad0817de2c887c3fb23ca0d33abd333d50ba164c96adc5875177bec945306218f4dccd57700244bf0b604ca2c4de5c9f9df8fbfe6faaa07f970c85881), signing_key: 7cede71d0c20bcc3014635c79b4ca0d33759a3340f5799cba9368eb9ddc58df7, slots: 231..284 }, Validator { address: Address("95496bf079f8b824e61e52e4eecd4426d6b8cde1"), voting_key: LazyPublicKey(c8512dc4a69729170fe2ac5cf6c9b25322fa974ceee569b5d4f86a5238a51de896deb5e6998d59f90944aad3e47e676433d13137b1bb0cffeeb158ce5594c937294a8330475b79a2463fa752b1fc851e42383175a7f05b0183475da14129013e5b437436f6705ef9e4509391663806ff62e7abf8a49e583c3a5741e2272f014f50f686f2181849f05d43d82ed5d62f69a427529b4f981ce582ad811565db5e1b18c6b7a6bf6664c6d54eaf7fd7424fd6a92f13543568681a4115b15204003f3efe58109efd1af4c591bb3984aac1e74eee4ddd5f5f4900d62feb2ac1e046ccf52c66d8526c69a6ca72823dcf3e060d7e488f7aded41d6f94bc9876a4083d243cff5fa6e4758674270d18ec9c32fbd553049bd0394da82cfa7a69390600), signing_key: 81def5effc93ae283b5acfa466d0a544218b1bf3c3558f6cf6403bd9aed14dee, slots: 284..369 }, Validator { address: Address("a86db9670a7c1a28f8377d850304379594e99e36"), voting_key: LazyPublicKey(53a4acb81010e714c8b3e85888a39bbb8252ea7edea3dee1b7962cf5c57f6bf1cd110d9b150b9531cf0eac428bbdeb34fec6d61a837dbcdec6624630341326ee04a58058f26e4f2df83e6732530729be189c5502d2f911a87eb738cd5bb2012e8990c98cc12966a056f2471109f134d13c62a4da99ba91de0bf15d42b0b5c03fe942e1f8107d371bdae866ed1f9af6ea76f0ed39605f53b8c3058b682718640f1cf16ecaf330177567bc75e07eec37e4fb8d4eca45feeba36a92f98d34006b54c5c8c3ca926b071a0910035983ec5d144ae03d6771b2dc25528e23615b046d63cf06241d79c4c081d1ae79346ca84062c9f84420f5ee918fdb3d511e29011d65be59bfc260073c328fe03746d7c5ee19498a23e5cea84705e91995fb00), signing_key: b5487c57e8f22f18968dd68c51671649a80b496266b3239353cd767c606692de, slots: 369..434 }, Validator { address: Address("ded24a92947a31ae4495f5f134dc82c5a29f76df"), voting_key: LazyPublicKey(e6459600e07c11f28eb225fb9b2ca3ae8aaab06835afc557087e39f6177a505482f00d908dc401d342e84cd83327fbaff578d9bb6264b39b5a52905c0cb37d6f255eb49c1cf66e7828a94e44c1d7433154e9cd7b646b16224ee79cfdaaed00424d176afe578b7c2643dd03e1fd237129db30f772368911847592781b4d382413a6800997ee71f16daca8fd04dc689ae77922cea72b8c64fa4bfd2d08fcd182fcf81297b0b3d760beeb1a49bef91375ddc6af74413c596b8f9fd8634c7300477b7fa471a73fbf2c4a4c30fbc207f927b3cae8212f3f2a46c382de933e84aa7810648243843bdc81a2ad807917533eda1111ba38a18f903097941ed59733f8d3f5e364fa8bc379285e703a9af636b0a17618d99a68413225941bd622f780), signing_key: fc2d2b8e3b7abfa39b79d2c2969c55a9996dbf79f6bd2fe80fad7c810ac279ea, slots: 434..435 }, Validator { address: Address("e00787fb0be4e48212660aeb205a8a76b78d0392"), voting_key: LazyPublicKey(db0a7ab07398f84802819ea9d2c365c39b115962b2e3c92b83c2b76fb9bfccc9e326b818035853bf2c6219c1be5bbedd730a1d5346341b28e90baba646cb7f8ea9f398edb959b0f8891839361bedd46c4439ce125c65b821d1970cfdd38e00ff098bc6ae624fe7d175f2305af9189e4ca3293cf1d280fdf78cde13c375ee3fd5c1e840c87a9b5230cc655d5ba5968f96401dcd057561a890ad910149fc935249706d9095bdf66b884682ed44d008e0e645dfd76fa2d29a513e58d3c16900851a702131bcf6710a852b1e60f05913f3537e49eeffa2584fad057b15abd01f65dd1e06fd996c6edcb43af6b9e0a9feafdf5dc2c4a636da2f9f505249ddccef17fb03975c125766ac72b3eb80d3f686444978b70efd24be8e33c70bdfa980), signing_key: 38a15d65eaef2462702332026dbd18d3f2e3c692e02427f3033cfd6f126762c3, slots: 435..457 }, Validator { address: Address("e346ce21e99068990c5df271ed0ed25e094d3156"), voting_key: LazyPublicKey(e9859318d2985998a8753ced371682cd3fc3638ab0d75028d39305625b844ea452bd00e39cfdec4bf48d8cbc2b1fade2eb1c555f30270a4d391dff9dc55985a09137bca7e15248f8f446af49ddd16562ca3c95232386d901a7ae51de1dc0010b8d9785b198c8781083d4b09aaf5d58ba24d9c2bd29d88fc54cfbd8053c41bb2041f7ef0328ff32906b77ab1664e707c4da4723acce5ec2c2df39f4698d4cdd77b82b3a282b0ebf0034bc0f4fc3d7c11fd00cc8533ecabbdcb7dcf27d0e019775ca127da296a62a430c417bc8fbd8fe2c1d0bee0e291b6adab795e4f802adb10c9d589c8b7a6b01810291e9fe2b1a226496cf768f0d35f135d03afafcf16e0e2a75702280299b481e2e0e5ad7e6cb58ced498d8cce76446882778568400), signing_key: 9c630945f2764818e8ea0924b8cea68a6680aa344edfd8103fd882392556ca03, slots: 457..512 }], validator_map: {Address("1dab7b1fc3483d8822eb223a3a84bf5a7a033081"): 0, Address("6c6c822f592ff5e7b07d5acb284ea282c54884ec"): 1, Address("7a493b7fa61f4ed323c67f94d25d95f5719bdc9c"): 2, Address("83cbb5be504fdde49fb2948476a7aa204933d205"): 3, Address("95496bf079f8b824e61e52e4eecd4426d6b8cde1"): 4, Address("a86db9670a7c1a28f8377d850304379594e99e36"): 5, Address("ded24a92947a31ae4495f5f134dc82c5a29f76df"): 6, Address("e00787fb0be4e48212660aeb205a8a76b78d0392"): 7, Address("e346ce21e99068990c5df271ed0ed25e094d3156"): 8} }, locators: [5711919193f2b22822c42a990200ad0adaba43737a5e378531cf56300339451f, 6b1e5d0e649d17e7112279614277c8dcb07aafb56807ee08fac9d4cb9733ec01, ea5dd050b281931bb74bd2f544fd30f667d52df4a00be6e0e52a3654c4955806], include_micro_bodies: false, direction: Forward }
2024-03-04T08:28:34.473878000Z DEBUG sync_queue           | Re-requesting MissingBlockRequest { target_block_number: 17778523, target_block_hash: a7cd50e34dcf087199386aac6a26a8753af7df72003ce1fa9c2f028a3651e314, epoch_validators: Validators { validators: [Validator { address: Address("1dab7b1fc3483d8822eb223a3a84bf5a7a033081"), voting_key: LazyPublicKey(7c127cda41181fc2fff5ef49b38d5b4b791aa02bac26bbd227f8564a26d38eb2619901b8bb465c00f2fe68b15c3cf26e59dbaea8cb07882065827950442c052ceeb171ae3e454acbc347aca42fc71d14ced130218e7761169bc5a9fd740501ca774db39197e2bc4df5853c4e354fc3f4804e9a96b1cc33e2720b314097ccb553439b6b33446d10b7f76b1c7ea90882d56fd30c58d9454589ce1dea6b9e8295c9de1c8c0e5c6615a8c1c188948f0d5d4dffbc842ff97b50b2fc2a77f07000b3a58025fddca83aca027166c92469f3b84c88cbc894e7fed33bc70b1bedfaa47e1050196a228fde5e1dbec94e48a907b15a72d5f84ef6afc098e29fcd626a9d6eaff12eee1774d0f420b7dc5be943b8cd4afea715da95b90f63343e217300), signing_key: d419c718543cbeb6068e322800d53aeb7bc90f221fcc92c80247ccd914ced438, slots: 0..65 }, Validator { address: Address("6c6c822f592ff5e7b07d5acb284ea282c54884ec"), voting_key: LazyPublicKey(81f8c6e4d629b91fae9afb4fcb4ce5639002cc5a27b10669752cae00d60879133681dc113e4ea5acc05e1a157b2bf07ddcbfa449a8999f9420cc4fc57bd25fdb23d564a5d9de0337853415342090c17d5131fad9c1c81e8632410b803d3b00baf9bd234d3510512665f083a644eca7bbe41fc1c06c1cead3d4e4f52550182af411e0cff067bb749d23e3d22784be671902813062f04f7877c682dd6039f954caea0038704e91614b0389b6ae584e81526f16040297dceaf4ba994af5d100543d87de2dc0c033449d706b6a4dad44ea90f49e7fcbd888dec4e95cc35efaf04108335ef74f97730dd0c809b25a737f7f9e821800e27b2e3078b65aa4af516adc649a514f6cb46981f3bf080f312a872e3a120b709905310ff3b75fe06681), signing_key: b1881f7b5412d456a9acb7d9021d3b032686608c407b487ff377d07cd9fbf244, slots: 65..168 }, Validator { address: Address("7a493b7fa61f4ed323c67f94d25d95f5719bdc9c"), voting_key: LazyPublicKey(fcdd7c2a13dfa03fb29ea92c8edbd4315055d84114f75a3bc36fc311e6eae698e9cc5d6c7e1e4a4a0f937a694fa666e47eb6f1a21d44a4f9b69f2acf6f25e3289d77c3bb3caab154dee0e263ec79c0a263e5c84c45c7e2d84c8f7f96f356015be91276a5d6e4923009ccc42fff3474b51609f31dff4410b25017a265dc350c698a06597391fb67a15003df808da1cdf70aa313c80cae0e793c6289d7e420ab4084acf32d531ebf2eafbe73e6b2ab923f3589efc75ba8e9bb0c62a7e11e00116e3f1a4164540b01073decb05ca56c83ec6eacb4af0234910b1b0133f06556003601e5e00d25d905c68c9fb95e3c28859cadd49b23a2a1b5606d63de7b2f1b8806badef3fb188d9be4cb9215926762b6cf0743d44c8790544f0a77be2701), signing_key: ee1424f128e4b8ae9a4951b7fb4717fa94cd9c74f29dbffdf591cdedbe149a8d, slots: 168..231 }, Validator { address: Address("83cbb5be504fdde49fb2948476a7aa204933d205"), voting_key: LazyPublicKey(23dede6c4eae710f2a278246391d01b9ff9e1e87987e72f367c99612a7779171843bb74437860a843869648f5967af9c06767f669cf9a80f8f57f7eff8692cd04b19fba5d3fd25f030bba3acf1417c5aef0a355d95297e91c9d18ea9e03b009144b235079487557e393e1a261c75c06dc1a92ae46a4fcbc15834766091fbde00143f0f8a884bb541a17ec644e92452a79ee96c31aed01396d2cd843ba91448d55570c6c1ea16e7296a29443d003c5daed6d51c2d58e4b89c791606fac7009c23773f0eda7a90ea68e615c136e4d75da11d5f28364038830a6cb2f16935217de02ad0817de2c887c3fb23ca0d33abd333d50ba164c96adc5875177bec945306218f4dccd57700244bf0b604ca2c4de5c9f9df8fbfe6faaa07f970c85881), signing_key: 7cede71d0c20bcc3014635c79b4ca0d33759a3340f5799cba9368eb9ddc58df7, slots: 231..284 }, Validator { address: Address("95496bf079f8b824e61e52e4eecd4426d6b8cde1"), voting_key: LazyPublicKey(c8512dc4a69729170fe2ac5cf6c9b25322fa974ceee569b5d4f86a5238a51de896deb5e6998d59f90944aad3e47e676433d13137b1bb0cffeeb158ce5594c937294a8330475b79a2463fa752b1fc851e42383175a7f05b0183475da14129013e5b437436f6705ef9e4509391663806ff62e7abf8a49e583c3a5741e2272f014f50f686f2181849f05d43d82ed5d62f69a427529b4f981ce582ad811565db5e1b18c6b7a6bf6664c6d54eaf7fd7424fd6a92f13543568681a4115b15204003f3efe58109efd1af4c591bb3984aac1e74eee4ddd5f5f4900d62feb2ac1e046ccf52c66d8526c69a6ca72823dcf3e060d7e488f7aded41d6f94bc9876a4083d243cff5fa6e4758674270d18ec9c32fbd553049bd0394da82cfa7a69390600), signing_key: 81def5effc93ae283b5acfa466d0a544218b1bf3c3558f6cf6403bd9aed14dee, slots: 284..369 }, Validator { address: Address("a86db9670a7c1a28f8377d850304379594e99e36"), voting_key: LazyPublicKey(53a4acb81010e714c8b3e85888a39bbb8252ea7edea3dee1b7962cf5c57f6bf1cd110d9b150b9531cf0eac428bbdeb34fec6d61a837dbcdec6624630341326ee04a58058f26e4f2df83e6732530729be189c5502d2f911a87eb738cd5bb2012e8990c98cc12966a056f2471109f134d13c62a4da99ba91de0bf15d42b0b5c03fe942e1f8107d371bdae866ed1f9af6ea76f0ed39605f53b8c3058b682718640f1cf16ecaf330177567bc75e07eec37e4fb8d4eca45feeba36a92f98d34006b54c5c8c3ca926b071a0910035983ec5d144ae03d6771b2dc25528e23615b046d63cf06241d79c4c081d1ae79346ca84062c9f84420f5ee918fdb3d511e29011d65be59bfc260073c328fe03746d7c5ee19498a23e5cea84705e91995fb00), signing_key: b5487c57e8f22f18968dd68c51671649a80b496266b3239353cd767c606692de, slots: 369..434 }, Validator { address: Address("ded24a92947a31ae4495f5f134dc82c5a29f76df"), voting_key: LazyPublicKey(e6459600e07c11f28eb225fb9b2ca3ae8aaab06835afc557087e39f6177a505482f00d908dc401d342e84cd83327fbaff578d9bb6264b39b5a52905c0cb37d6f255eb49c1cf66e7828a94e44c1d7433154e9cd7b646b16224ee79cfdaaed00424d176afe578b7c2643dd03e1fd237129db30f772368911847592781b4d382413a6800997ee71f16daca8fd04dc689ae77922cea72b8c64fa4bfd2d08fcd182fcf81297b0b3d760beeb1a49bef91375ddc6af74413c596b8f9fd8634c7300477b7fa471a73fbf2c4a4c30fbc207f927b3cae8212f3f2a46c382de933e84aa7810648243843bdc81a2ad807917533eda1111ba38a18f903097941ed59733f8d3f5e364fa8bc379285e703a9af636b0a17618d99a68413225941bd622f780), signing_key: fc2d2b8e3b7abfa39b79d2c2969c55a9996dbf79f6bd2fe80fad7c810ac279ea, slots: 434..435 }, Validator { address: Address("e00787fb0be4e48212660aeb205a8a76b78d0392"), voting_key: LazyPublicKey(db0a7ab07398f84802819ea9d2c365c39b115962b2e3c92b83c2b76fb9bfccc9e326b818035853bf2c6219c1be5bbedd730a1d5346341b28e90baba646cb7f8ea9f398edb959b0f8891839361bedd46c4439ce125c65b821d1970cfdd38e00ff098bc6ae624fe7d175f2305af9189e4ca3293cf1d280fdf78cde13c375ee3fd5c1e840c87a9b5230cc655d5ba5968f96401dcd057561a890ad910149fc935249706d9095bdf66b884682ed44d008e0e645dfd76fa2d29a513e58d3c16900851a702131bcf6710a852b1e60f05913f3537e49eeffa2584fad057b15abd01f65dd1e06fd996c6edcb43af6b9e0a9feafdf5dc2c4a636da2f9f505249ddccef17fb03975c125766ac72b3eb80d3f686444978b70efd24be8e33c70bdfa980), signing_key: 38a15d65eaef2462702332026dbd18d3f2e3c692e02427f3033cfd6f126762c3, slots: 435..457 }, Validator { address: Address("e346ce21e99068990c5df271ed0ed25e094d3156"), voting_key: LazyPublicKey(e9859318d2985998a8753ced371682cd3fc3638ab0d75028d39305625b844ea452bd00e39cfdec4bf48d8cbc2b1fade2eb1c555f30270a4d391dff9dc55985a09137bca7e15248f8f446af49ddd16562ca3c95232386d901a7ae51de1dc0010b8d9785b198c8781083d4b09aaf5d58ba24d9c2bd29d88fc54cfbd8053c41bb2041f7ef0328ff32906b77ab1664e707c4da4723acce5ec2c2df39f4698d4cdd77b82b3a282b0ebf0034bc0f4fc3d7c11fd00cc8533ecabbdcb7dcf27d0e019775ca127da296a62a430c417bc8fbd8fe2c1d0bee0e291b6adab795e4f802adb10c9d589c8b7a6b01810291e9fe2b1a226496cf768f0d35f135d03afafcf16e0e2a75702280299b481e2e0e5ad7e6cb58ced498d8cce76446882778568400), signing_key: 9c630945f2764818e8ea0924b8cea68a6680aa344edfd8103fd882392556ca03, slots: 457..512 }], validator_map: {Address("1dab7b1fc3483d8822eb223a3a84bf5a7a033081"): 0, Address("6c6c822f592ff5e7b07d5acb284ea282c54884ec"): 1, Address("7a493b7fa61f4ed323c67f94d25d95f5719bdc9c"): 2, Address("83cbb5be504fdde49fb2948476a7aa204933d205"): 3, Address("95496bf079f8b824e61e52e4eecd4426d6b8cde1"): 4, Address("a86db9670a7c1a28f8377d850304379594e99e36"): 5, Address("ded24a92947a31ae4495f5f134dc82c5a29f76df"): 6, Address("e00787fb0be4e48212660aeb205a8a76b78d0392"): 7, Address("e346ce21e99068990c5df271ed0ed25e094d3156"): 8} }, locators: [5711919193f2b22822c42a990200ad0adaba43737a5e378531cf56300339451f, 6b1e5d0e649d17e7112279614277c8dcb07aafb56807ee08fac9d4cb9733ec01, ea5dd050b281931bb74bd2f544fd30f667d52df4a00be6e0e52a3654c4955806], include_micro_bodies: false, direction: Forward } @ 129 peer_id=12D3KooWGbKfQvV4r59BCYHoRzGbARTRFsxbx5SpUQb8C3ciQSiM current_peer_index=2
2024-03-04T08:28:34.530582000Z DEBUG block_request_compo… | Received empty missing blocks response

Crash log? Screenshots? Videos? Sample project?

light-client-falling-behind.log

[Eligioo]

Looking deeper into this it is interesting that requesting missing block 17778525 seems to succeed. However for 17778523 this request seems to be failing even after multiple attempts with Received empty missing blocks response as reason. Block 17778524 is a macro block which sits just between 23 and 25.

Possible theory: other nodes are unable to locate the requested block 17778523 since the head is 17778526 which is already in the new batch.

However this failing shouldn't lock-up the whole queue which let me think that somewhere there is a reference to block 17778523 kept and since processing these blocks are ordered, it keeps processing this as first.

[nibhar]

Your assessment is correct.

Specifically a request to a block with the oldest locators in the future of that block will return an empty response. The oldest possible locator to any given head height is always the most recent macro block.

Generally such requests should never be send, as it can be determined before sending it that the response will be empty, making it pointless. On the other hand it should also not prohibit progress if they are.

[Eligioo]

Exactly @nibhar, just came to the same conclusion. The direction of the MissingBlockRequest is Forward with locators of the new batch. This block will never be found this way.

[nibhar]

It should also no longer be necessary as a more recent macro block has been adopted.

[nibhar]

Ok I am fairly confident that I have found the cause to this.

Let's assume #10 and #7 are macro blocks. Our Node is a Light Node.

To begin with our Node knows block number #9:MI. It then sees block #11:MI over gossip. If will then end up here in the BlockQueue and ultimately in that function it will buffer and request the missing blocks here.
That will create a request for the missing block #10:MA with the macro height of #7:MA. That does make sense and that is in fact a missing block and that request can also be resolved.

Before that request resolves the node sees #10:MA on gossip. It knows the predecessor and the block gets pushed. By virtue of being a light node the chain store gets pruned such that only #10:MA remains (see here).

Now the block queue can happily push block #11:MI and so on.

Now the Missing blocks request from earlier resolves and we get #10:MA from that. Requested blocks have a different code path than gossipped blocks and they start here.

Now within the handle_missing_blocks function call the order of checks is as such that the block #10:MA is the target hash of the request but its predecessor is not known.

That creates another missing bocks request. This time for #9:MI with the macro head of the now known #10:MA. That request does not make sense and it is also not resolvable by the server side of the request.

[nibhar]

The fix should be as easy as adding a sanity check before the predecessor check into the handle_missing_blocks function to make sure a received block is in the current batch.

I.e.

// Check if the block is still relevant if not discard it.
if self.blockchain.read().macro_head().block_number() >=  target_block_number {
   return None;
}

I will try and see if I can create a test for this first though.

[nibhar]

Proposed fix on this branch nibhar/handle-missing-blocks. As I said, I will first try to also get a test together for this.