Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mainnet state difference at *some* of the RPC nodes at *empty* block 5282529 #3424

Closed
AnnaShaleva opened this issue Apr 27, 2024 · 9 comments · Fixed by #3443
Closed

Mainnet state difference at *some* of the RPC nodes at *empty* block 5282529 #3424

AnnaShaleva opened this issue Apr 27, 2024 · 9 comments · Fixed by #3443
Assignees
@AnnaShaleva
Labels
bug Something isn't working I4 No visible changes S4 Routine U1 Critically important to resolve quickly
Milestone
v0.106.0

Comments

@AnnaShaleva
Copy link
Member

Broken RPC:

anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ go run scripts/compare-states/compare-states.go http://seed2.neo.org:10332 https://rpc1.n3.nspcc.ru:10331
at 0: 58a5157b7e99eeabf631291f1747ec8eb12ab89461cda888492b17a301de81e8 vs 58a5157b7e99eeabf631291f1747ec8eb12ab89461cda888492b17a301de81e8
at 5282588: 98a1824895f48c34ecf501d0bc8a6a018cc403c32131606f3dd92108668410e1 vs 7e21b8e163eae5debe6542f8f41964db388a42add53c176a1e29ba87a638772a
at 2641294: 3eb50728cd6ab5aa0e5b940418e7b8ca864b8a2810fcbc39efb3cdc005c358de vs 3eb50728cd6ab5aa0e5b940418e7b8ca864b8a2810fcbc39efb3cdc005c358de
at 3961941: 8ea86f266267a43f4c91c5d823e99e247a540d84b4cd954cdfc012d7ad56899e vs 8ea86f266267a43f4c91c5d823e99e247a540d84b4cd954cdfc012d7ad56899e
at 4622264: 051d577cfe55d920f5d782a310dd3d059dc3f3c197a8c56dd7e07cdbc87c5324 vs 051d577cfe55d920f5d782a310dd3d059dc3f3c197a8c56dd7e07cdbc87c5324
at 4952426: 28a5fd25c001b1102edb5072b4eebb6e5a47827b252d08427e6042cbba1f4159 vs 28a5fd25c001b1102edb5072b4eebb6e5a47827b252d08427e6042cbba1f4159
at 5117507: d2fe26ea73795f911a520797424692a80d928c86fcee190a96951553a54e2eef vs d2fe26ea73795f911a520797424692a80d928c86fcee190a96951553a54e2eef
at 5200047: 24bafe80302b1dc5b092ce39d78412115a55df06cbc98c1ded99c26731061c99 vs 24bafe80302b1dc5b092ce39d78412115a55df06cbc98c1ded99c26731061c99
at 5241317: 6f97ddebc5f96e69303c5381a5c220c22bf8af088b737084aff1c25dc31ffb43 vs 6f97ddebc5f96e69303c5381a5c220c22bf8af088b737084aff1c25dc31ffb43
at 5261952: e1421243b269b2fdf1b43629288095d05dcb2ab8e69a4e987a392ffb8824ef1f vs e1421243b269b2fdf1b43629288095d05dcb2ab8e69a4e987a392ffb8824ef1f
at 5272270: fd9187451c9a7f2183eccb5e2dfd0e9810efd57f13a97e6c0b2e4ae3aade379c vs fd9187451c9a7f2183eccb5e2dfd0e9810efd57f13a97e6c0b2e4ae3aade379c
at 5277429: f9d10ac488c4dadff663832c2d09a71bbb01a3e2627d705f9777df6d9d82a757 vs f9d10ac488c4dadff663832c2d09a71bbb01a3e2627d705f9777df6d9d82a757
at 5280008: 6510e92bd6a1f6c27734e3c9b6cc0b479c7fc263f0dbc2b9aa14ac5cf8d29081 vs 6510e92bd6a1f6c27734e3c9b6cc0b479c7fc263f0dbc2b9aa14ac5cf8d29081
at 5281298: b35840f8bd3997b880569ee175633bf1e6ef93bea807a4bbafd9c0d6598a4c54 vs b35840f8bd3997b880569ee175633bf1e6ef93bea807a4bbafd9c0d6598a4c54
at 5281943: 5ab276a0d8fb1c3257c6d98d1cc05601056cafd2f635e607dd16daf329c3d20d vs 5ab276a0d8fb1c3257c6d98d1cc05601056cafd2f635e607dd16daf329c3d20d
at 5282265: b9761a5af66fba3b8000cf021b2bb86fdaac40059ae51931dbe502beb8f77662 vs b9761a5af66fba3b8000cf021b2bb86fdaac40059ae51931dbe502beb8f77662
at 5282426: 6766dbede7f44a63484b542b01456d4b3604c8e4cdff29e8da454de54f97c94d vs 6766dbede7f44a63484b542b01456d4b3604c8e4cdff29e8da454de54f97c94d
at 5282507: 7f59c47c43ca1124dd39c594e27c28ea4a9f4747998d5f6014200a4464521228 vs 7f59c47c43ca1124dd39c594e27c28ea4a9f4747998d5f6014200a4464521228
at 5282547: a12e88d419d0d6d9daa37a378de17544265aa3d069f1626c0b4f6ed96dd60bdc vs c37fbf76d263cac1c081d020309fa6546f16385136b83dd6ad5aa9f80e52d5a8
at 5282527: 883f2a8114880dd8a5a7f8c642fc547b238910b67d95263214321bd5f92eb39f vs 883f2a8114880dd8a5a7f8c642fc547b238910b67d95263214321bd5f92eb39f
at 5282537: ec01cfbdb130e63c7ad7ebc127e9b6d35f9d5391fc2fe62d53bdf1d3e4b6dad3 vs e0c1047791dc957272ce513311c9e3abc562dc70fffc143e03d94ec55f9afd6e
at 5282532: 8859cd3da2364feff4a2fc96909fca3541f9766ffb8886932f0e55ad38e096cc vs bd9f3d7f928c8ec27b0ffa4561f21f80d250b8fbafbe0f1f7be7ab7e84d9e676
at 5282529: 5f4007fe0d488ec6a31a7c0dc2d58473f58940e260ca4c20bc766206a075a58f vs 29fed48978bd6fa46cd6c2debdb3cc75ba2cf10d075601fbab5dd765db5fd25e
at 5282528: 7edeb900368986a0317654b90f68d7aaf51323d20f1218064a820aee384b7d83 vs 7edeb900368986a0317654b90f68d7aaf51323d20f1218064a820aee384b7d83
state differs at 5282529, block 4eb55e069672795b8744efc75e9992e4a1575ecac34720e0c42c227c72681750
different state found
exit status 1

Good RPC:

anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ go run scripts/compare-states/compare-states.go http://seed2.neo.org:10332 https://rpc10.n3.nspcc.ru:10331
at 0: 58a5157b7e99eeabf631291f1747ec8eb12ab89461cda888492b17a301de81e8 vs 58a5157b7e99eeabf631291f1747ec8eb12ab89461cda888492b17a301de81e8
at 5282586: ca34437912335a13125c520979267064e702ee921b0da0ab830d6246079b1037 vs ca34437912335a13125c520979267064e702ee921b0da0ab830d6246079b1037

Might be related to #3273.
@AnnaShaleva AnnaShaleva added bug Something isn't working U2 Seriously planned I2 Regular impact U1 Critically important to resolve quickly S1 Highly significant and removed U2 Seriously planned labels Apr 27, 2024
@AnnaShaleva AnnaShaleva added this to the v0.106.0 milestone Apr 27, 2024
@AnnaShaleva AnnaShaleva self-assigned this Apr 27, 2024
@roman-khimov roman-khimov added S4 Routine I4 No visible changes and removed I2 Regular impact S1 Highly significant labels Apr 27, 2024
@roman-khimov
Copy link
Member

More fun, rpc2 (but others are fine again):

at 0: 58a5157b7e99eeabf631291f1747ec8eb12ab89461cda888492b17a301de81e8 vs 58a5157b7e99eeabf631291f1747ec8eb12ab89461cda888492b17a301de81e8
at 5288082: 97a0d105736bce02079987d46b0a0f7bf8e490c873d223539a8df127f998c18c vs ccf35bf35813b60f939376d6adcb34ad63c2bcf7e02822bc113257342bc23a29
at 2644041: 5c567e182ab23be49025d27b59549b975ccb673d7cf03429ea612d2be5973306 vs 5c567e182ab23be49025d27b59549b975ccb673d7cf03429ea612d2be5973306
at 3966061: 5039735233ed1058f602757b1dde7faee9e38104a169e4616c0e451d99b3cef2 vs 5039735233ed1058f602757b1dde7faee9e38104a169e4616c0e451d99b3cef2
at 4627071: 66a77f08900417f1fc487b409c59f684adce60b52a1c48c72ff0b7a7de4e5284 vs 66a77f08900417f1fc487b409c59f684adce60b52a1c48c72ff0b7a7de4e5284
at 4957576: 3e70e9a460480049e9545d167303008b06b356087c2d030d50816b2add93c2b6 vs 3e70e9a460480049e9545d167303008b06b356087c2d030d50816b2add93c2b6
at 5122829: f074add0cd793436b8e845df648956ec209701fdbac47620f00736a101d88c01 vs f074add0cd793436b8e845df648956ec209701fdbac47620f00736a101d88c01
at 5205455: bc4d63d5b85eb7df84e4cc9f46b3f0a5ae22710aff0bba8fd38ec5e8e08c905d vs bc4d63d5b85eb7df84e4cc9f46b3f0a5ae22710aff0bba8fd38ec5e8e08c905d
at 5246768: 8b4b26946c773a35aabb771bde54dddcd68cc2760386b372032f0c1b30ded3ac vs 8b4b26946c773a35aabb771bde54dddcd68cc2760386b372032f0c1b30ded3ac
at 5267425: 2e84cb5dcdf55a5c2130602561fcda4e66a80d13f303e180afaef25c05171d21 vs 2e84cb5dcdf55a5c2130602561fcda4e66a80d13f303e180afaef25c05171d21
at 5277753: 53c9a752f0634b9b48b04d8b388b5fa49d41162a2a674abdea9080217a2d4b7d vs 53c9a752f0634b9b48b04d8b388b5fa49d41162a2a674abdea9080217a2d4b7d
at 5282917: 40d9e9f08fcb72e29b79bf062d60883011d8f1075f8fc29c3daad07731cbac4d vs 40d9e9f08fcb72e29b79bf062d60883011d8f1075f8fc29c3daad07731cbac4d
at 5285499: 1f787037583a78ac087e3fbf48a2860b765f546467823d3980569e08326e09cf vs 1f787037583a78ac087e3fbf48a2860b765f546467823d3980569e08326e09cf
at 5286790: c2d8a81b8845eade1c1fdc1ba99e3179e69900e5c74cb4623f5bcc31e6624de4 vs c2d8a81b8845eade1c1fdc1ba99e3179e69900e5c74cb4623f5bcc31e6624de4
at 5287436: c7311e3a2f59e5dcb777fdfbc12b4a69e940fa7a3679a72eb7624bba492c0c0c vs c7311e3a2f59e5dcb777fdfbc12b4a69e940fa7a3679a72eb7624bba492c0c0c
at 5287759: 69bdfd4238553144decb57c9fa439c94cd12f07f36b4309c4e1ff5a4e7e870d9 vs 69bdfd4238553144decb57c9fa439c94cd12f07f36b4309c4e1ff5a4e7e870d9
at 5287920: ee03cca02db4c3ea8fbb8e38a04ac88bf26a2181601cc2d659e3fbba7f604a8d vs ee03cca02db4c3ea8fbb8e38a04ac88bf26a2181601cc2d659e3fbba7f604a8d
at 5288001: 412cda87f85c5fba0cbe67ed959443fffe7d03b7c114c47f44ebf3bc9bd88a1d vs da876651649028b6ab5f59e3a87ae0535f82b4db11c051a9211f7e81bd001699
at 5287960: 1301abf36f1fdc60af37c5a4661e08ab73821c61fa4b24ab7c45d8621de3bd93 vs 1301abf36f1fdc60af37c5a4661e08ab73821c61fa4b24ab7c45d8621de3bd93
at 5287980: 69d51d9725b0f99cd8bd8aae74e3f03d25e815bed4e1a3aee1f45acf54cce143 vs 69d51d9725b0f99cd8bd8aae74e3f03d25e815bed4e1a3aee1f45acf54cce143
at 5287990: 0e1cde6a1c6145cb2f18413e0cf46188193f18c2890e136c3270d0ec2f58af6e vs 6cf8e3f1789acb5e80aaf25a6aa03d167801b71a418832b4e990dbbc09253332
at 5287985: bd8522b5ebfe22b0d9777108f9622a26570931952f338d6ac74eba46952abada vs bd8522b5ebfe22b0d9777108f9622a26570931952f338d6ac74eba46952abada
at 5287987: 02f563c0e9dbe78d31231befd4330df184036f6162a8614f0d2489d547f583ad vs 02f563c0e9dbe78d31231befd4330df184036f6162a8614f0d2489d547f583ad
at 5287988: 10ca652bd0ce46c40c99028123274e27f7c0b3135aa73412ce8dc15dadb82e9e vs 10ca652bd0ce46c40c99028123274e27f7c0b3135aa73412ce8dc15dadb82e9e
at 5287989: eb98bf0e698b86ab216e83847a6834c128ce0f707b1eec4e0a7c454d691bda97 vs d7db68dedc9e2987697717f98c04eec8e5a6703a327bb4c291a694a02726eedf
state differs at 5287989, block b6abe1b2054a85de4659c8d2082e94c7d2bea4e1a99a8c210ea939c8d79b0079
transaction 949581fd8240bf41520a07274fae7d1ed4b3f5856d1d9c386bcd8b68e19d9cc6:
--- https://rpc2.n3.nspcc.ru:10331
+++ http://seed3.neo.org:10332
@@ -1,2 +1,2 @@
-(*result.ApplicationLog)(0xc0000a1380)({
+(*result.ApplicationLog)(0xc0002bf980)({
  Container: (util.Uint256) (len=32 cap=32) c69c9de1688bcd6b389c1d6d85f5b3d41e7dae4f27070a5241bf4082fd819594,
@@ -15,3 +15,3 @@
      Name: (string) (len=8) "Transfer",
-     Item: (*stackitem.Array)(0xc000408fc0)(Array)
+     Item: (*stackitem.Array)(0xc0001ffbf0)(Array)
     }

different state found
exit status 1

AnnaShaleva added a commit that referenced this issue May 14, 2024
@AnnaShaleva
scripts: dump diff for OnPersist and PostPersist applog
0a3bf01 
Related to #3424.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
@AnnaShaleva AnnaShaleva mentioned this issue May 14, 2024
Merged
@AnnaShaleva
Copy link
Member Author

More fun, rpc2 (but others are fine again:
transaction 949581fd8240bf41520a07274fae7d1ed4b3f5856d1d9c386bcd8b68e19d9cc6:

This one strange, I've checked the transaction logs for rpc2 and seed3 manually, and they are identical:
Response from http://seed3.neo.org:10332:

anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ curl -d '{ "jsonrpc": "2.0", "id": 1, "method": "getapplicationlog", "params": ["949581fd8240bf41520a07274fae7d1ed4b3f5856d1d9c386bcd8b68e19d9cc6"] }' http://seed3.neo.org:10332 | json_pp
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   688    0   548  100   140    653    167 --:--:-- --:--:-- --:--:--   821
{
   "id" : 1,
   "jsonrpc" : "2.0",
   "result" : {
      "executions" : [
         {
            "exception" : null,
            "gasconsumed" : "1592356",
            "notifications" : [
               {
                  "contract" : "0xf0151f528127558851b39c2cd8aa47da7418ab28",
                  "eventname" : "Transfer",
                  "state" : {
                     "type" : "Array",
                     "value" : [
                        {
                           "type" : "ByteString",
                           "value" : "rkZWKw9dFazBTho/SyKtqDVLfjM="
                        },
                        {
                           "type" : "ByteString",
                           "value" : "uvnIKaLr79Lm4EcIVw3HHJYhhS0="
                        },
                        {
                           "type" : "Integer",
                           "value" : "1100000000"
                        }
                     ]
                  }
               }
            ],
            "stack" : [
               {
                  "type" : "Boolean",
                  "value" : true
               }
            ],
            "trigger" : "Application",
            "vmstate" : "HALT"
         }
      ],
      "txid" : "0x949581fd8240bf41520a07274fae7d1ed4b3f5856d1d9c386bcd8b68e19d9cc6"
   }
}

Response from https://rpc2.n3.nspcc.ru:10331:

anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ curl -d '{ "jsonrpc": "2.0", "id": 1, "method": "getapplicationlog", "params": ["949581fd8240bf41520a07274fae7d1ed4b3f5856d1d9c386bcd8b68e19d9cc6"] }' https://rpc2.n3.nspcc.ru:10331 | json_pp
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   689  100   549  100   140   5202   1326 --:--:-- --:--:-- --:--:--  6561
{
   "id" : 1,
   "jsonrpc" : "2.0",
   "result" : {
      "executions" : [
         {
            "exception" : null,
            "gasconsumed" : "1592356",
            "notifications" : [
               {
                  "contract" : "0xf0151f528127558851b39c2cd8aa47da7418ab28",
                  "eventname" : "Transfer",
                  "state" : {
                     "type" : "Array",
                     "value" : [
                        {
                           "type" : "ByteString",
                           "value" : "rkZWKw9dFazBTho/SyKtqDVLfjM="
                        },
                        {
                           "type" : "ByteString",
                           "value" : "uvnIKaLr79Lm4EcIVw3HHJYhhS0="
                        },
                        {
                           "type" : "Integer",
                           "value" : "1100000000"
                        }
                     ]
                  }
               }
            ],
            "stack" : [
               {
                  "type" : "Boolean",
                  "value" : true
               }
            ],
            "trigger" : "Application",
            "vmstate" : "HALT"
         }
      ],
      "txid" : "0x949581fd8240bf41520a07274fae7d1ed4b3f5856d1d9c386bcd8b68e19d9cc6"
   }
}

But there are OnPersist/PostPersist differences as far in this block, and these differences are real:

anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ go run scripts/compare-states/compare-states.go --ignore-height http://seed3.neo.org:10332 https://rpc2.n3.nspcc.ru:10331
at 0: 58a5157b7e99eeabf631291f1747ec8eb12ab89461cda888492b17a301de81e8 vs 58a5157b7e99eeabf631291f1747ec8eb12ab89461cda888492b17a301de81e8
at 5288114: ad33398822c20315ac92b515198fdc94012967ed2eef0bb6800e7ec31279d2b0 vs 005a8ede1fe0235d04d296b836274f50a3b2fb6caa13fe111255b206b829aa07
at 2644057: 631ace74e7dc5f9407075e8f90a6e28c870401af40107f5d87ec0c37bdc450b9 vs 631ace74e7dc5f9407075e8f90a6e28c870401af40107f5d87ec0c37bdc450b9
at 3966085: 3555026f97371420084623647b5785da61e3cee55b242eae85fb88101bba32f4 vs 3555026f97371420084623647b5785da61e3cee55b242eae85fb88101bba32f4
at 4627099: 9fd96c6ac31f92f0361d359432f03c939997cf5fff5b390c70a23a5114fcde19 vs 9fd96c6ac31f92f0361d359432f03c939997cf5fff5b390c70a23a5114fcde19
at 4957606: ff8ede6ed30cb3abe5aa46131fbf4fb2f01bdeb0b51cc51047cfabb151480ec8 vs ff8ede6ed30cb3abe5aa46131fbf4fb2f01bdeb0b51cc51047cfabb151480ec8
at 5122860: 58f27e0a286885e444d255a6d69600d90243546b93a6045b19426037e4e6a758 vs 58f27e0a286885e444d255a6d69600d90243546b93a6045b19426037e4e6a758
at 5205487: 30fd7f2b8239fa29c9c79a8334b4cd13a2c7c802909b70bf00a4d78789973f6b vs 30fd7f2b8239fa29c9c79a8334b4cd13a2c7c802909b70bf00a4d78789973f6b
at 5246800: 1d01a8df23a3e0f53bbc14bb7d3e5259d6641d6ea1a04f999cc50fc71bffc2e7 vs 1d01a8df23a3e0f53bbc14bb7d3e5259d6641d6ea1a04f999cc50fc71bffc2e7
at 5267457: b514b77a34a65fa2fc35fc9bf5b76dec7d0b4d07c7ac9e6ce8060756dd0c982a vs b514b77a34a65fa2fc35fc9bf5b76dec7d0b4d07c7ac9e6ce8060756dd0c982a
at 5277785: 1ca7657be25f472211ae45703f8d86b1b8f26f069844c4ed5bc430127387b151 vs 1ca7657be25f472211ae45703f8d86b1b8f26f069844c4ed5bc430127387b151
at 5282949: 48dc5d7f4dcf4af6aab4e29baeec865e9ec23212264a0bd2bfae20572f7164ea vs 48dc5d7f4dcf4af6aab4e29baeec865e9ec23212264a0bd2bfae20572f7164ea
at 5285531: 18221e997aaf02a04f93220627aa3c41bfe8b107cf19194ec7d21bdd22b0b8a1 vs 18221e997aaf02a04f93220627aa3c41bfe8b107cf19194ec7d21bdd22b0b8a1
at 5286822: 0b419c83a0f27277529f97aa36ef7ec5887694730f26fe2728bec576cd2b3b4b vs 0b419c83a0f27277529f97aa36ef7ec5887694730f26fe2728bec576cd2b3b4b
at 5287468: 6f24ad5c157071ec3c0c1126ffb156a84ae49ad796fb813f91818580cad6e329 vs 6f24ad5c157071ec3c0c1126ffb156a84ae49ad796fb813f91818580cad6e329
at 5287791: c33ed03a717dd6165d78094a7b747f99cb5f72ce99434f619f0e42bacc27eff4 vs c33ed03a717dd6165d78094a7b747f99cb5f72ce99434f619f0e42bacc27eff4
at 5287952: ceec5539e77c95a68bbca83b484c8c4a76031f1a07b7416fa38b677549315028 vs ceec5539e77c95a68bbca83b484c8c4a76031f1a07b7416fa38b677549315028
at 5288033: 98ca54df0637d028da8e2f75f6f104061b45627bd7a326da1b0717dcb39a351c vs dd8a25fe7083731dc6f276187d415097fc2fd00af275126c65fe0a082bd84a5c
at 5287992: a60719a3bc8689cabfb2552e5626342ee4497f9bdb841017e127d5e4f2aaaf71 vs 5cf3c93018c52131d0accd5d565154d4550330e4586cf52fd9f3c299efa5ae63
at 5287972: cf03c36374e7d2b14b429d00d9a1e8007f739c6e62df55c9c018aac8c13dc69c vs cf03c36374e7d2b14b429d00d9a1e8007f739c6e62df55c9c018aac8c13dc69c
at 5287982: a2430ee62f066d9f6433ab03e38f9d8e9548992bdfb8fc1f9f55201d863a444f vs a2430ee62f066d9f6433ab03e38f9d8e9548992bdfb8fc1f9f55201d863a444f
at 5287987: 02f563c0e9dbe78d31231befd4330df184036f6162a8614f0d2489d547f583ad vs 02f563c0e9dbe78d31231befd4330df184036f6162a8614f0d2489d547f583ad
at 5287989: d7db68dedc9e2987697717f98c04eec8e5a6703a327bb4c291a694a02726eedf vs eb98bf0e698b86ab216e83847a6834c128ce0f707b1eec4e0a7c454d691bda97
at 5287988: 10ca652bd0ce46c40c99028123274e27f7c0b3135aa73412ce8dc15dadb82e9e vs 10ca652bd0ce46c40c99028123274e27f7c0b3135aa73412ce8dc15dadb82e9e
state differs at 5287989, block b6abe1b2054a85de4659c8d2082e94c7d2bea4e1a99a8c210ea939c8d79b0079
block b6abe1b2054a85de4659c8d2082e94c7d2bea4e1a99a8c210ea939c8d79b0079:
--- http://seed3.neo.org:10332
+++ https://rpc2.n3.nspcc.ru:10331
@@ -1,2 +1,2 @@
-(*result.ApplicationLog)(0xc0004055c0)({
+(*result.ApplicationLog)(0xc0005363c0)({
  Container: (util.Uint256) (len=32 cap=32) 79009bd7c839a90e218c9aa9e1a4bed2c7942e08d2c85946de854a05b2e1abb6,
@@ -14,3 +14,3 @@
      Name: (string) (len=8) "Transfer",
-     Item: (*stackitem.Array)(0xc0002c97d0)(Array)
+     Item: (*stackitem.Array)(0xc000410c90)(Array)
     },
@@ -19,3 +19,3 @@
      Name: (string) (len=8) "Transfer",
-     Item: (*stackitem.Array)(0xc0002c9a10)(Array)
+     Item: (*stackitem.Array)(0xc000410ed0)(Array)
     }
@@ -34,3 +34,3 @@
      Name: (string) (len=8) "Transfer",
-     Item: (*stackitem.Array)(0xc0002c9cb0)(Array)
+     Item: (*stackitem.Array)(0xc000411170)(Array)
     }

transaction 949581fd8240bf41520a07274fae7d1ed4b3f5856d1d9c386bcd8b68e19d9cc6:
--- http://seed3.neo.org:10332
+++ https://rpc2.n3.nspcc.ru:10331
@@ -1,2 +1,2 @@
-(*result.ApplicationLog)(0xc000405840)({
+(*result.ApplicationLog)(0xc000404040)({
  Container: (util.Uint256) (len=32 cap=32) c69c9de1688bcd6b389c1d6d85f5b3d41e7dae4f27070a5241bf4082fd819594,
@@ -15,3 +15,3 @@
      Name: (string) (len=8) "Transfer",
-     Item: (*stackitem.Array)(0xc00045e2a0)(Array)
+     Item: (*stackitem.Array)(0xc0001e6570)(Array)
     }

different state found
exit status 1

@AnnaShaleva
Copy link
Member Author

Some more info about sorting order of validators/committee:

RPC4 (https://rpc4.n3.nspcc.ru:10331), block 5282529:

Candidates sorted by votes, top 21:

NSPCCpw8YmgNDYWiBfXJHRfz38NDjv6WW3: 9966524  <--- should receive reward for block
NZeAarn3UMCqNsTymTMF2Pn6X7Yw3GhqDv: 3021664  <--- chosen by RPC4 node
NiYfNbJXhHs9WvuP2PWR5RFR9VCjdGn69w: 2164726
Nhvpo1kz1iv8KuBB1KGAbUxHet4V1Gzz4u: 2157272
NTE8wUDSXVk7oqbG1kZKTxSPX5Xj2nsLjd: 2061789
Nj39M97Rk2e23JiULBBMQmvpcnKaRHqxFf: 2055593
NYz4EgdsM1ATNedAbxFJw499kDBWhc8uut: 2039642
---
NV17k94y5JS4mBjETmeKyHs3y3kxEfiRsM: 2007915
NhMvRrhBnZyAwZnw8y9mHoCzwSEDmZJo2n: 1001190
NZ9rkPKcDQqH6bffyYqU6yd5A2cUvuDLUw: 685698 
NSTSntFPK36QXsjEK6oAhnPzSyfgfVA2GQ: 666925 
NbobgCMmKhHFfdfAfL7579W4bHrsh66wPt: 659066 
NhUHywGfUYevMjtbpRDDgGBgkwzYiGD9W9: 648939 
NSZunX9DXXvdMrjQqQWqGAMvgNTPsbhQ2L: 646393 
NiymdbpxariiQ5kW16TnAkPdPwd9KNJUBT: 643287 
NhkVeDbeZc6So1LbzhAt9KNH8A9yuPEhDS: 642806 
NZ6bKQGT6mWqbXRNjX9ohAr5fVZwifWtGW: 642336 
NaxjMApXDkYQc3paoyAErLsm6XWo1FUw4X: 640595 
NaqNz8kAYezSMUnK6CZxduc7oFxDaYZeCb: 640350 
Nc8EimiHubCEX329XAnVFsS15yzwdnRYj1: 636986 
NcScdqRaoE6DVzvGDBAnias9GTivdWfrDf: 635853

Committee sorted by pubs:

NhUHywGfUYevMjtbpRDDgGBgkwzYiGD9W9
NbobgCMmKhHFfdfAfL7579W4bHrsh66wPt
Nj39M97Rk2e23JiULBBMQmvpcnKaRHqxFf
NaxjMApXDkYQc3paoyAErLsm6XWo1FUw4X
NTE8wUDSXVk7oqbG1kZKTxSPX5Xj2nsLjd
NSTSntFPK36QXsjEK6oAhnPzSyfgfVA2GQ
NcScdqRaoE6DVzvGDBAnias9GTivdWfrDf
NZ9rkPKcDQqH6bffyYqU6yd5A2cUvuDLUw
NaqNz8kAYezSMUnK6CZxduc7oFxDaYZeCb
NhMvRrhBnZyAwZnw8y9mHoCzwSEDmZJo2n
NSZunX9DXXvdMrjQqQWqGAMvgNTPsbhQ2L
NiYfNbJXhHs9WvuP2PWR5RFR9VCjdGn69w
NV17k94y5JS4mBjETmeKyHs3y3kxEfiRsM
NZeAarn3UMCqNsTymTMF2Pn6X7Yw3GhqDv
Nhvpo1kz1iv8KuBB1KGAbUxHet4V1Gzz4u
Nc8EimiHubCEX329XAnVFsS15yzwdnRYj1
NYz4EgdsM1ATNedAbxFJw499kDBWhc8uut
NiymdbpxariiQ5kW16TnAkPdPwd9KNJUBT
NSPCCpw8YmgNDYWiBfXJHRfz38NDjv6WW3
NZ6bKQGT6mWqbXRNjX9ohAr5fVZwifWtGW
NhkVeDbeZc6So1LbzhAt9KNH8A9yuPEhDS

RPC2 (https://rpc2.n3.nspcc.ru:10331), block 5287989:

Candidates sorted by votes, top 21:

NSPCCpw8YmgNDYWiBfXJHRfz38NDjv6WW3: 9966533  <--- should be block reward receiver
NZeAarn3UMCqNsTymTMF2Pn6X7Yw3GhqDv: 3021664  <--- chosen by RPC2 node
NiYfNbJXhHs9WvuP2PWR5RFR9VCjdGn69w: 2165228
Nhvpo1kz1iv8KuBB1KGAbUxHet4V1Gzz4u: 2157272
NTE8wUDSXVk7oqbG1kZKTxSPX5Xj2nsLjd: 2061789
Nj39M97Rk2e23JiULBBMQmvpcnKaRHqxFf: 2055593
NYz4EgdsM1ATNedAbxFJw499kDBWhc8uut: 2039642
---
NV17k94y5JS4mBjETmeKyHs3y3kxEfiRsM: 2007915
NhMvRrhBnZyAwZnw8y9mHoCzwSEDmZJo2n: 1001190
NZ9rkPKcDQqH6bffyYqU6yd5A2cUvuDLUw: 679305 
NSTSntFPK36QXsjEK6oAhnPzSyfgfVA2GQ: 667133 
NbobgCMmKhHFfdfAfL7579W4bHrsh66wPt: 659066 
NhUHywGfUYevMjtbpRDDgGBgkwzYiGD9W9: 648008 
NSZunX9DXXvdMrjQqQWqGAMvgNTPsbhQ2L: 647618 
NiymdbpxariiQ5kW16TnAkPdPwd9KNJUBT: 643287 
Nc8EimiHubCEX329XAnVFsS15yzwdnRYj1: 642408 
NaxjMApXDkYQc3paoyAErLsm6XWo1FUw4X: 640160 
NcScdqRaoE6DVzvGDBAnias9GTivdWfrDf: 640139 
NaqNz8kAYezSMUnK6CZxduc7oFxDaYZeCb: 640115 
NZ6bKQGT6mWqbXRNjX9ohAr5fVZwifWtGW: 638705 
NhkVeDbeZc6So1LbzhAt9KNH8A9yuPEhDS: 638343

Validators sorted by default primary index:

validator 0: Nj39M97Rk2e23JiULBBMQmvpcnKaRHqxFf  <--- should be fees receiver (CN reward receiver)
validator 1: NTE8wUDSXVk7oqbG1kZKTxSPX5Xj2nsLjd  <--- chosen by RPC2 node
validator 2: NiYfNbJXhHs9WvuP2PWR5RFR9VCjdGn69w
validator 3: NZeAarn3UMCqNsTymTMF2Pn6X7Yw3GhqDv
validator 4: Nhvpo1kz1iv8KuBB1KGAbUxHet4V1Gzz4u
validator 5: NYz4EgdsM1ATNedAbxFJw499kDBWhc8uut
validator 6: NSPCCpw8YmgNDYWiBfXJHRfz38NDjv6WW3

Committee sorted by pubs:

NhUHywGfUYevMjtbpRDDgGBgkwzYiGD9W9
NbobgCMmKhHFfdfAfL7579W4bHrsh66wPt
Nj39M97Rk2e23JiULBBMQmvpcnKaRHqxFf
NaxjMApXDkYQc3paoyAErLsm6XWo1FUw4X
NTE8wUDSXVk7oqbG1kZKTxSPX5Xj2nsLjd
NSTSntFPK36QXsjEK6oAhnPzSyfgfVA2GQ
NcScdqRaoE6DVzvGDBAnias9GTivdWfrDf
NZ9rkPKcDQqH6bffyYqU6yd5A2cUvuDLUw
NaqNz8kAYezSMUnK6CZxduc7oFxDaYZeCb
NhMvRrhBnZyAwZnw8y9mHoCzwSEDmZJo2n
NSZunX9DXXvdMrjQqQWqGAMvgNTPsbhQ2L
NiYfNbJXhHs9WvuP2PWR5RFR9VCjdGn69w
NV17k94y5JS4mBjETmeKyHs3y3kxEfiRsM
NZeAarn3UMCqNsTymTMF2Pn6X7Yw3GhqDv
Nhvpo1kz1iv8KuBB1KGAbUxHet4V1Gzz4u
Nc8EimiHubCEX329XAnVFsS15yzwdnRYj1
NYz4EgdsM1ATNedAbxFJw499kDBWhc8uut
NiymdbpxariiQ5kW16TnAkPdPwd9KNJUBT
NSPCCpw8YmgNDYWiBfXJHRfz38NDjv6WW3
NZ6bKQGT6mWqbXRNjX9ohAr5fVZwifWtGW
NhkVeDbeZc6So1LbzhAt9KNH8A9yuPEhDS

@roman-khimov
Copy link
Member

Notice that for block 5282530 on RPC4:

         {
            "stack" : [],
            "gasconsumed" : "0",
            "notifications" : [
               {
                  "state" : {
                     "value" : [
                        {
                           "type" : "Any"
                        },
                        {
                           "value" : "e6Kzuk8PbRM81V2ZgHneG4KX4zo=",
                           "type" : "ByteString"
                        },
                        {
                           "type" : "Integer",
                           "value" : "50000000"
                        }
                     ],
                     "type" : "Array"
                  },
                  "eventname" : "Transfer",
                  "contract" : "0xd2a4cff31913016155e38e474a2c06d08be276cf"
               }
            ],
            "vmstate" : "HALT",
            "trigger" : "PostPersist",
            "exception" : null
         }

It's NXBhD662PnMFHZ1jJnreVTx71tdmqtrjL9 and it's completely missing from the list. But 5282531:

         {
            "notifications" : [
               {
                  "contract" : "0xd2a4cff31913016155e38e474a2c06d08be276cf",
                  "state" : {
                     "type" : "Array",
                     "value" : [
                        {
                           "type" : "Any"
                        },
                        {
                           "value" : "8XwthjF7tAPBIE7duMRv00rHcKs=",
                           "type" : "ByteString"
                        },
                        {
                           "type" : "Integer",
                           "value" : "50000000"
                        }
                     ]
                  },
                  "eventname" : "Transfer"
               }
            ],
            "exception" : null,
            "stack" : [],
            "trigger" : "PostPersist",
            "gasconsumed" : "0",
            "vmstate" : "HALT"
         }

is Nhvpo1kz1iv8KuBB1KGAbUxHet4V1Gzz4u.

@roman-khimov
Copy link
Member

Then it's NYz4EgdsM1ATNedAbxFJw499kDBWhc8uut and then NXsJYaejf5EFrFgSuPp4XUXajQ8BXUVoN8.

NXsJYaejf5EFrFgSuPp4XUXajQ8BXUVoN8 and NXBhD662PnMFHZ1jJnreVTx71tdmqtrjL9 look like voters for validators.

@roman-khimov
Copy link
Member

Except they don't really vote.

@roman-khimov
Copy link
Member

It's NF2/NF5 nodes, from down below of the https://governance.neo.org/

@roman-khimov
Copy link
Member 

NZeAarn3UMCqNsTymTMF2Pn6X7Yw3GhqDv
NXBhD662PnMFHZ1jJnreVTx71tdmqtrjL9
Nhvpo1kz1iv8KuBB1KGAbUxHet4V1Gzz4u
NYz4EgdsM1ATNedAbxFJw499kDBWhc8uut
NXsJYaejf5EFrFgSuPp4XUXajQ8BXUVoN8

is exactly

  StandbyCommittee:
  - 03b209fd4f53a7170ea4444e0cb0a6bb6a53c2bd016926989cf85f9b0fba17a70c
  - 02df48f60e8f3e01c48ff40b9b7f1310d7a8b2a193188befe1c2e3df740e895093
  - 03b8d9d5771d8f513aa0869b9cc8d50986403b78c6da36890638c3d46a5adce04a
  - 02ca0e27697b9c248f6f16e085fd0061e26f44da85b58ee835c110caa5ec3ba554
  - 024c7b7fb6c310fccf1ba33b082519d82964ea93868d676662d4a59ad548df0e7d

@roman-khimov
Copy link
Member

So it looks like voterTurnout.Sign() != 1 || len(cs) < count. But it's hard to tell now since

$ ./bin/neo-go contract testinvokefunction --historic 5282529 -r https://rpc2.n3.nspcc.ru:10331 0xef4073a0f2b305a38ec4050e4d3d28bc40ea63f5 getCandidates 
Old state requests are not supported (-606) - only latest state is supported: 'KeepOnlyLatestState' setting is enabled

AnnaShaleva added a commit that referenced this issue May 17, 2024
@AnnaShaleva
native: fix Policy's IsBlocked behaviour
2da8b6b 
Account is blocked when it's in the Policy's storage. Introduced in
bbbc680.

This bug leads to the fact that during native Neo cache initialization
at the last block in the dBFT epoch, all candidates accounts are
"blocked", and thus, stand-by committee and validators are used in the
subsequent new epoch. Close #3424.

This bug may lead to the consequenses described in #3273, but it needs
to be confirmed.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
AnnaShaleva added a commit that referenced this issue May 17, 2024
@AnnaShaleva
native: fix Policy's IsBlocked behaviour
4ee3c6d 
Account is blocked when it's in the Policy's storage, not when it's
missing from the Policy storage. Introduced in
bbbc680.

This bug leads to the fact that during native Neo cache initialization
at the last block in the dBFT epoch, all candidates accounts are
"blocked", and thus, stand-by committee and validators are used in the
subsequent new epoch. Close #3424.

This bug may lead to the consequenses described in #3273, but it needs
to be confirmed.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
AnnaShaleva added a commit that referenced this issue May 17, 2024
@AnnaShaleva
native: fix Policy's IsBlocked behaviour
c43cfae 
Account is blocked when it's in the Policy's storage, not when it's
missing from the Policy storage. Introduced in
bbbc680.

This bug leads to the fact that during native Neo cache initialization
at the last block in the dBFT epoch, all candidates accounts are
"blocked", and thus, stand-by committee and validators are used in the
subsequent new epoch. Close #3424.

This bug may lead to the consequences described in #3273, but it needs
to be confirmed.

Signed-off-by: Anna Shaleva <shaleva.ann@nspcc.ru>
@AnnaShaleva AnnaShaleva mentioned this issue May 17, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@AnnaShaleva AnnaShaleva

Labels
bug Something isn't working I4 No visible changes S4 Routine U1 Critically important to resolve quickly
Projects
None yet
Milestone
v0.106.0
Development

Successfully merging a pull request may close this issue.

native: fix Policy's IsBlocked behaviour nspcc-dev/neo-go
2 participants
@roman-khimov @AnnaShaleva