Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bridge nodes listener gets stuck until resub #3339

Open
walldiss opened this issue Apr 25, 2024 · 3 comments
Open

Bridge nodes listener gets stuck until resub #3339

walldiss opened this issue Apr 25, 2024 · 3 comments
Assignees
@walldiss
Labels
area:core_and_app Relationship with Core node and Celestia-App bug Something isn't working

Comments

@walldiss
Copy link
Member

Implementation ideas

Initially reported by the P-OPS team, this issue was noted when bridge nodes occasionally fell behind consensus nodes. The issue is consistently marked by the following log line:

listener: subscriber error, resubscribing...

Upon further investigation across other bridge nodes, similar occurrences were noted a few times each week. These issues are generally resolved automatically by the node's internal logic, which triggers a resubscription in response to such events.
Notably, these events affect multiple nodes simultaneously, which suggests potential connectivity issues at the time of the incident or a problem with the core node RPC becoming stuck.
@walldiss walldiss added the enhancement New feature or request label Apr 25, 2024
@walldiss walldiss self-assigned this Apr 25, 2024
@walldiss walldiss added bug Something isn't working area:core_and_app Relationship with Core node and Celestia-App and removed enhancement New feature or request labels Apr 25, 2024
@malise800
Copy link

hey @walldiss so i will put here information that seems relevant to the issue. hope that will help.

so today we had the issue for around 20 min
image

issue was resolved with a manual restart

here are the logs captured just before the issue and after

2024-04-25T07:51:00.825Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "80A22C1AC58BFB6482A0ECFB8DD045C971A98FC0B13E5779459AFB93C1545A6E"}
2024-04-25T07:51:09.589Z        INFO    header/store    store/store.go:375      new head        {"height": 1685431, "hash": "F6B9A16D324B16E492DA9F2B3F8A2D0E32691EF41FB056D6F887D7CAE9A73C6E"}
2024-04-25T07:51:16.904Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "8CA74145AD1A865FB99727428D03EC0166D995725574A860053D56861DA77C37"}
2024-04-25T07:51:21.544Z        INFO    header/store    store/store.go:375      new head        {"height": 1685432, "hash": "D284D7162C514311161AFE9DD6CB6E55A7BE6BF901A88A36AB065DA2ABC23D7C"}
2024-04-25T07:51:22.336Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T07:51:27.981Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "C36C693ABD7E239572F93481C9D01111C4CBCB775B4E219AFCEB196476844A02"}
2024-04-25T07:51:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460       Block cache might be too small. Metrics: hit: 2432177 miss: 61534984 keys-added: 5820577 keys-updated: 13703 keys-evicted: 5820556 cost-added: 60101415840754 cost-evicted: 60101154390680 sets-dropped: 191 sets-rejected: 48316623 gets-dropped: 4414016 gets-kept: 59550016 gets-total: 63967161 hit-ratio: 0.04
2024-04-25T07:51:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461       Cache life expectancy (in seconds):
 -- Histogram:
Min value: 0
Max value: 20
Count: 5820366
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 5814781 99.90% 99.90%
[2, 4) 4497 0.08% 99.98%
[4, 8) 1004 0.02% 100.00%
[8, 16) 83 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
 --
2024-04-25T07:51:35.936Z        INFO    canonical-log   swarm/swarm_dial.go:559 CANONICAL_PEER_STATUS: peer=12D3KooWEjMK8hk451SvUixvSC3eA4Mj2jqxwhgTvuQGJ71wcV1r addr=/ip6/2607:5300:203:6730::/udp/2121/quic-v1 sample_rate=100 connection_status="established" dir="outbound"
2024-04-25T07:51:41.668Z        ERROR   shrex/eds       shrexeds/server.go:120  server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "DD8A7EA64D5BC8DA35A9607B63886E7375F619BA1BE5B12682281822A71FE0B4", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to acquire reader of mount so we can return the accessor: open /home/pops/.celestia-bridge-mocha-4/blocks/DD8A7EA64D5BC8DA35A9607B63886E7375F619BA1BE5B12682281822A71FE0B4: too many open files"}
2024-04-25T07:51:42.114Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:51:55.177Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "B79369019A087FD68B234E81E24B7EA9CB5905572262B5571352ACD342C6EDEB"}
2024-04-25T07:52:09.108Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "158E8D9F71F7EE6E2923782118656C09F132BD9A8C709DAAD1456E241639DE74"}
2024-04-25T07:52:11.544Z        INFO    core    core/listener.go:201    listener: listening stopped
2024-04-25T07:52:11.545Z        WARN    core    core/listener.go:134    listener: subscriber error, resubscribing...    {"err": "underlying subscription is stuck"}
2024-04-25T07:52:18.977Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "268B0C7C095F81FC23F0CFD47601628C60CD472F1540883E1DEF8712BD5EFE42"}
2024-04-25T07:52:23.875Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "D406C8E570566EBAE40558B66559529F08D08EC343412D547A9CBB5FB4183693"}
2024-04-25T07:52:28.815Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "8C6A1B85FC532538E7300E6F0371E1007633C38C7F8E6C0C588698B3DD9A4376"}
2024-04-25T07:52:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460       Block cache might be too small. Metrics: hit: 2446119 miss: 61823594 keys-added: 5846729 keys-updated: 13773 keys-evicted: 5846716 cost-added: 60383210996091 cost-evicted: 60382948333249 sets-dropped: 191 sets-rejected: 48579011 gets-dropped: 4414016 gets-kept: 59852672 gets-total: 64269713 hit-ratio: 0.04
2024-04-25T07:52:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461       Cache life expectancy (in seconds):
 -- Histogram:
Min value: 0
Max value: 20
Count: 5846526
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 5840888 99.90% 99.90%
[2, 4) 4539 0.08% 99.98%
[4, 8) 1013 0.02% 100.00%
[8, 16) 85 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
 --
2024-04-25T07:52:49.450Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "25ECCF519AB6A8A6967F34B90933B6A7E101E014BA87BAC824D322FBD08BC7CA"}
2024-04-25T07:52:52.111Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:53:01.185Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "BC75CA8425AFD344C0E2AE42EA9DF96D9AED2ACE809EE4C8F7D9F010E3BED11E"}
2024-04-25T07:53:16.177Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "1D9B5337C3A6A2BD0CE7E484D0324567E62D50BB8A5F28DD1860131727184AAB"}
2024-04-25T07:53:17.593Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:53:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460       Block cache might be too small. Metrics: hit: 2456866 miss: 62052959 keys-added: 5868417 keys-updated: 13824 keys-evicted: 5868394 cost-added: 60622560334831 cost-evicted: 60622294449708 sets-dropped: 191 sets-rejected: 48786637 gets-dropped: 4414016 gets-kept: 60092864 gets-total: 64509825 hit-ratio: 0.04
2024-04-25T07:53:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461       Cache life expectancy (in seconds):
 -- Histogram:
Min value: 0
Max value: 20
Count: 5868204
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 5862515 99.90% 99.90%
[2, 4) 4584 0.08% 99.98%
[4, 8) 1019 0.02% 100.00%
[8, 16) 85 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
 --
2024-04-25T07:53:42.662Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:53:44.113Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "C93A7F6AF89660C45430B2789E26422962A1913899275DFFEE433E4E24DA7F19"}
2024-04-25T07:53:58.154Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "257997AAA44BBC5A996FC94E106738910EAEFE5D32AB3CB3240121238039E342"}
2024-04-25T07:54:22.114Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:54:32.000Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "1DFDC94B427F670BF162D61F5C4A3CB3F3B634221986EDB6C09F8B6E98B3DEFC"}
2024-04-25T07:54:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460       Block cache might be too small. Metrics: hit: 2468315 miss: 62299488 keys-added: 5892560 keys-updated: 13873 keys-evicted: 5892545 cost-added: 60863255870186 cost-evicted: 60862998787177 sets-dropped: 191 sets-rejected: 49008974 gets-dropped: 4414080 gets-kept: 60350784 gets-total: 64767803 hit-ratio: 0.04
2024-04-25T07:54:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461       Cache life expectancy (in seconds):
 -- Histogram:
Min value: 0
Max value: 20
Count: 5892355
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 5886608 99.90% 99.90%
[2, 4) 4626 0.08% 99.98%
[4, 8) 1033 0.02% 100.00%
[8, 16) 87 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
 --
2024-04-25T07:54:37.059Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T07:54:46.385Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "AB57DD8617CE7FC583A661AD48425933061428350CF52F93952F8CDA1E94E5F5"}
2024-04-25T07:54:54.872Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "4C8052FE35825456B4DFCF5D725A678B8D72617C81ABA15CDD753DD77789DDF9"}
2024-04-25T07:55:04.001Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "897194E2B7739FB971830B2A12BECA1B8439593DB51B14F29A774181F02ECD2B"}
2024-04-25T07:55:28.386Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "C76938FABE7F1E7CD35D60A25CCA7763692A3C70B26486C63D22DDE50E98CEA1"}
2024-04-25T07:55:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460       Block cache might be too small. Metrics: hit: 2476202 miss: 62476577 keys-added: 5910384 keys-updated: 13896 keys-evicted: 5910366 cost-added: 61065462695834 cost-evicted: 61065204139792 sets-dropped: 191 sets-rejected: 49168213 gets-dropped: 4414144 gets-kept: 60535616 gets-total: 64952780 hit-ratio: 0.04
2024-04-25T07:55:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461       Cache life expectancy (in seconds):
 -- Histogram:
Min value: 0
Max value: 20
Count: 5910175
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 5904324 99.90% 99.90%
[2, 4) 4714 0.08% 99.98%
[4, 8) 1047 0.02% 100.00%
[8, 16) 89 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
 --
2024-04-25T07:55:36.367Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "B41752E35B81BDE6BEDD0105AEC2323B5A2D28A0CF3A618299641DBD9BED54FC"}
2024-04-25T07:55:41.221Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "2917D2836C625BBEC179A8B9732FF2D5026C89DCFCB4C1E9DCDD86361E3B33E2"}
2024-04-25T07:55:42.112Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:55:42.662Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:55:45.811Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "39F656BAE5D7CB42F43FDC4E5F188BDDE8A38C2AF3659FBAF773EA953F96E1C8"}
2024-04-25T07:55:52.050Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "81F884A07C5D404B676D221036FABB7E231EDC235D886B0F4BEC4C4C72D8A4EB"}
2024-04-25T07:55:58.376Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "8EB8023C472AD5E5C0C2D718C32C548CC18B7B47131A41D0CE13397B1F31EF08"}
2024-04-25T07:56:02.809Z        ERROR   shrex/eds       shrexeds/server.go:120  server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "06E27242189AF9FAC12F75E6F412084CDE93F902D10BD5F576E9F3788FB38439", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to recover index for shard 06E27242189AF9FAC12F75E6F412084CDE93F902D10BD5F576E9F3788FB38439: open /home/pops/.celestia-bridge-mocha-4/index/06E27242189AF9FAC12F75E6F412084CDE93F902D10BD5F576E9F3788FB38439.full.idx: too many open files"}
2024-04-25T07:56:04.946Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "1359FEF60DF42A8E9BBEE34323C00D1EF42C2B15584AA442D2692B281269647E"}
2024-04-25T07:56:17.114Z        ERROR   shrex/eds       shrexeds/server.go:120  server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "4576AC0D9BAC9B60071E8E8ED971067EDFF5A804DD640FC1F29EAD4B9971FC70", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to acquire reader of mount so we can return the accessor: open /home/pops/.celestia-bridge-mocha-4/blocks/4576AC0D9BAC9B60071E8E8ED971067EDFF5A804DD640FC1F29EAD4B9971FC70: too many open files"}
2024-04-25T07:56:19.601Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "9E54C93D31AE2CBF40852EDEE90B181FEB374DAFD6BEF27054A748CFD4AEE189"}
2024-04-25T07:56:32.614Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "39F86185E4246BEF6BF1F1568C77D9BBEE935D70F4E697541E063B6A8F2E584F"}
2024-04-25T07:56:35.058Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460       Block cache might be too small. Metrics: hit: 2486870 miss: 62702951 keys-added: 5934641 keys-updated: 13952 keys-evicted: 5934610 cost-added: 61315199697615 cost-evicted: 61314935808776 sets-dropped: 191 sets-rejected: 49370276 gets-dropped: 4414208 gets-kept: 60772672 gets-total: 65189821 hit-ratio: 0.04
2024-04-25T07:56:35.058Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461       Cache life expectancy (in seconds):
 -- Histogram:
Min value: 0
Max value: 20
Count: 5934420
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 5928529 99.90% 99.90%
[2, 4) 4751 0.08% 99.98%
[4, 8) 1049 0.02% 100.00%
[8, 16) 90 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
 --
2024-04-25T07:56:43.369Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "144078B15D706E5D42CBEA2EA2A0BAC4CB9D1ABF54D692495AEAE6B3BE6CAFB9"}
2024-04-25T07:56:56.418Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "761E626C6DADC536989C4F0E8BE63ED84F447565323A08FB62C088B793F11174"}
2024-04-25T07:57:09.014Z        ERROR   shrex/eds       shrexeds/server.go:120  server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "6FCA2825072472A6C125CD871D5AC10A727B245509D4A2381FE52E2F3784BB10", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to recover index for shard 6FCA2825072472A6C125CD871D5AC10A727B245509D4A2381FE52E2F3784BB10: open /home/pops/.celestia-bridge-mocha-4/index/6FCA2825072472A6C125CD871D5AC10A727B245509D4A2381FE52E2F3784BB10.full.idx: too many open files"}
2024-04-25T07:57:22.116Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:57:31.559Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "7AAF0551B755CAEF7BE63981CECF6E052508400D7DAB647F3F700C69AFEC48F2"}
2024-04-25T07:57:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460       Block cache might be too small. Metrics: hit: 2496805 miss: 62909162 keys-added: 5955219 keys-updated: 13983 keys-evicted: 5955198 cost-added: 61505584066530 cost-evicted: 61505320080300 sets-dropped: 191 sets-rejected: 49555878 gets-dropped: 4414272 gets-kept: 60988800 gets-total: 65405967 hit-ratio: 0.04
2024-04-25T07:57:35.058Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461       Cache life expectancy (in seconds):
 -- Histogram:
Min value: 0
Max value: 20
Count: 5955008
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 5949006 99.90% 99.90%
[2, 4) 4836 0.08% 99.98%
[4, 8) 1072 0.02% 100.00%
[8, 16) 93 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
 --
2024-04-25T07:57:41.946Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "89C48C59D7F981818727D038A9475DA4CA26F6C4671CC4D23EFF4A98C83B518E"}
2024-04-25T07:57:42.662Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:58:12.181Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "8C15835E0E3406B5184B112241745B76B85166010E01CC983C1D476988F7AF46"}
2024-04-25T07:58:16.513Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "69E784EC92386B991240EA8C6FAB8978E2A4D771EB74134903E608643A8B0B78"}
2024-04-25T07:58:16.768Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T07:58:27.983Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "867F0F7CF371E8D4F262796751581F4F139A003E1FF413AD0A20ABC122A1E935"}
2024-04-25T07:58:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460       Block cache might be too small. Metrics: hit: 2508064 miss: 63143410 keys-added: 5980931 keys-updated: 14041 keys-evicted: 5980912 cost-added: 61776633321108 cost-evicted: 61776372161892 sets-dropped: 191 sets-rejected: 49764356 gets-dropped: 4414400 gets-kept: 61234176 gets-total: 65651474 hit-ratio: 0.04
2024-04-25T07:58:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461       Cache life expectancy (in seconds):
 -- Histogram:
Min value: 0
Max value: 20
Count: 5980722
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 5974689 99.90% 99.90%
[2, 4) 4866 0.08% 99.98%
[4, 8) 1073 0.02% 100.00%
[8, 16) 93 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
 --
2024-04-25T07:58:40.547Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "C86830074C2E67E22D9F2A3C2C88B2F30064874F339A1D76E4036F09C6BE84D4"}
2024-04-25T07:58:42.114Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:58:48.817Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "152B4FF3FCC5B8184AE6AD8F7A36CA5D1A72BD13BEED68CAAE98DED0457E3DE9"}
2024-04-25T07:59:13.033Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "1FE8AB4F4E3CF679A096ACBEC0F95003E40318E3F86097B2B61ABE4DEA031A2F"}
2024-04-25T07:59:21.503Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "1E71F32AB19F7B0BC42852DC8FD621C007AC7A7A441D8FFE9A25F86CA8836A97"}
2024-04-25T07:59:27.322Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "B61E769A505647830555B0393C0F3997862A0627642001CFC04A5C1DEC140F21"}
2024-04-25T07:59:35.058Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460       Block cache might be too small. Metrics: hit: 2517378 miss: 63339663 keys-added: 6002317 keys-updated: 14085 keys-evicted: 6002304 cost-added: 61993113001270 cost-evicted: 61992845144346 sets-dropped: 191 sets-rejected: 49939179 gets-dropped: 4414464 gets-kept: 61439552 gets-total: 65857041 hit-ratio: 0.04
2024-04-25T07:59:35.058Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461       Cache life expectancy (in seconds):
 -- Histogram:
Min value: 0
Max value: 20
Count: 6002114
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 5995993 99.90% 99.90%
[2, 4) 4934 0.08% 99.98%
[4, 8) 1091 0.02% 100.00%
[8, 16) 95 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
 --
2024-04-25T07:59:42.663Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T07:59:47.250Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "8669CFB0C8FED83F5811CB7B7E7A833CCFF43FE525598F0598B677AB51692C1F"}
2024-04-25T07:59:58.667Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "1A40C1C5F70C43B799396D7CA8B2F45E77152C5F045DD4730C1DAF73EB255D5C"}
2024-04-25T08:00:04.435Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "E2475D5C780A4D870B6F21F9A3FE9DC36D8AA3217F0955737301A510FB244C10"}
2024-04-25T08:00:09.504Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "E346C56C569B3543C9823051CC9F72D817770F0CB2965E36042788D8D8508B38"}
2024-04-25T08:00:15.436Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "7B80F9156F03F6C2921CEB6048BF8857AF4C333B310B977F074A615BFCC61DE5"}
2024-04-25T08:00:22.114Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:00:23.813Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "54D2BCA4072515039B19E60E2DE14F1768A656EDE819A144324E6B0C28758622"}
2024-04-25T08:00:25.780Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "499F39C762BC2CD9F72223E354470CA61DA942BA4CD84A05FE77F929E297841E"}
2024-04-25T08:00:25.881Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEhsc7GmWo9VcexP4SnRymojTaWqQUM7RxVF111bZxVp1", "hash": "499F39C762BC2CD9F72223E354470CA61DA942BA4CD84A05FE77F929E297841E"}
2024-04-25T08:00:26.106Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWNEDvy2uYwBrtic5BzfDioKHDdSoTC72jwzHewNoVv1B3", "hash": "499F39C762BC2CD9F72223E354470CA61DA942BA4CD84A05FE77F929E297841E"}
2024-04-25T08:00:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460       Block cache might be too small. Metrics: hit: 2529947 miss: 63602091 keys-added: 6029169 keys-updated: 14141 keys-evicted: 6029156 cost-added: 62265398471922 cost-evicted: 62265131402218 sets-dropped: 191 sets-rejected: 50174699 gets-dropped: 4414464 gets-kept: 61714624 gets-total: 66132038 hit-ratio: 0.04
2024-04-25T08:00:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461       Cache life expectancy (in seconds):
 -- Histogram:
Min value: 0
Max value: 20
Count: 6028966
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6022828 99.90% 99.90%
[2, 4) 4949 0.08% 99.98%
[4, 8) 1093 0.02% 100.00%
[8, 16) 95 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
 --
2024-04-25T08:00:38.421Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWNEDvy2uYwBrtic5BzfDioKHDdSoTC72jwzHewNoVv1B3", "hash": "4EE5E09736A4F410D671B817F3BFB19296798F2468838585EE1F944919A6A27A"}
2024-04-25T08:00:39.792Z        ERROR   shrex/eds       shrexeds/server.go:120  server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "34EF2288D12F5970BA428390A0B0AE8F864F5E2C0C45F6C231BC1DEFE44FF864", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to acquire reader of mount so we can return the accessor: open /home/pops/.celestia-bridge-mocha-4/blocks/34EF2288D12F5970BA428390A0B0AE8F864F5E2C0C45F6C231BC1DEFE44FF864: too many open files"}
2024-04-25T08:00:49.823Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "6078DA78BC0ED4B04B73DA0AC8DB546889C30E52446FDDC6FA6B99F5A3FCFB4B"}
2024-04-25T08:00:49.966Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEhsc7GmWo9VcexP4SnRymojTaWqQUM7RxVF111bZxVp1", "hash": "195F34CF7E89AD076F1B7B42B862A2523D305A36E1B98737F8A68DA8011F9EF3"}
2024-04-25T08:00:53.703Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "6F2F6C8F9F8BA70DF4D10014DEACEA1598F011013918320DBB48B40A4787B8DF"}
2024-04-25T08:00:57.296Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "5A57563C32F522FD389F1A5EDA13BC1EFB16E3A963BA013F85AEAD7303F1EF6C"}
2024-04-25T08:01:03.737Z        ERROR   shrex/eds       shrexeds/server.go:120  server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "C4DF337245B41821F21CF8EA8CEF923F7A790C0987B50E156B9F88958F06E791", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to acquire reader of mount so we can return the accessor: open /home/pops/.celestia-bridge-mocha-4/blocks/C4DF337245B41821F21CF8EA8CEF923F7A790C0987B50E156B9F88958F06E791: too many open files"}
2024-04-25T08:01:06.758Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "80EC66847272995B42EC1B3226DE0922C15137002758715A45EE377330B9C97E"}
2024-04-25T08:01:10.575Z        ERROR   shrex/eds       shrexeds/server.go:120  server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "D38ECBD79DECD642F5D109571773DC16F77B35E19B9D6BBF7497756033C4F664", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to acquire reader of mount so we can return the accessor: open /home/pops/.celestia-bridge-mocha-4/blocks/D38ECBD79DECD642F5D109571773DC16F77B35E19B9D6BBF7497756033C4F664: too many open files"}
2024-04-25T08:01:13.099Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "6F245B89A9BB52FB0B19E899C9DC7D26DB40AE8C3F23AE3437586C695E6759F6"}
2024-04-25T08:01:17.353Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "26BEAFA950AC9CAC56A844491AE269F7AD2A8450B8847F2D875B7BFB02846C35"}
2024-04-25T08:01:24.559Z        ERROR   shrex/eds       shrexeds/server.go:120  server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "28F7B44497104B64BBFC9C17CD6D1FC2862215867D37BDC3DC8E172585CB44AC", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to acquire reader of mount so we can return the accessor: open /home/pops/.celestia-bridge-mocha-4/blocks/28F7B44497104B64BBFC9C17CD6D1FC2862215867D37BDC3DC8E172585CB44AC: too many open files"}
2024-04-25T08:01:25.152Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEhsc7GmWo9VcexP4SnRymojTaWqQUM7RxVF111bZxVp1", "hash": "01864E5FA7FB0DA4A0AEDA1EC256EE8EFB41CE91C6C0F39180CA5220EA420F54"}
2024-04-25T08:01:30.081Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "2803D7651F963BFCDECF1DB043AEAC4F56194726DD316F5E448AA221652491B6"}
2024-04-25T08:01:32.115Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:01:35.058Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460       Block cache might be too small. Metrics: hit: 2540507 miss: 63817284 keys-added: 6050291 keys-updated: 14178 keys-evicted: 6050268 cost-added: 62484802899070 cost-evicted: 62484545874197 sets-dropped: 191 sets-rejected: 50368733 gets-dropped: 4414464 gets-kept: 61940480 gets-total: 66357791 hit-ratio: 0.04
2024-04-25T08:01:35.058Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461       Cache life expectancy (in seconds):
 -- Histogram:
Min value: 0
Max value: 20
Count: 6050078
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6043853 99.90% 99.90%
[2, 4) 5018 0.08% 99.98%
[4, 8) 1109 0.02% 100.00%
[8, 16) 97 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
 --
2024-04-25T08:01:42.235Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "F403F032A1EA7D44DD223806F18F1AB74202971B9F5A72D956635AA0570A487D"}
2024-04-25T08:01:42.662Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:01:45.058Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:01:45.398Z        INFO    canonical-log   swarm/swarm_listen.go:136       CANONICAL_PEER_STATUS: peer=12D3KooWJ5dfgoAXpG2hsvXhtxTCp5BsxEWkzGWhbmns8zzf9Z2a addr=/ip4/54.191.40.227/udp/2121/quic-v1 sample_rate=100 connection_status="established" dir="inbound"
2024-04-25T08:01:52.571Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:02:02.691Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "2356198DF2B17027119A58F5098D5300AA47A6E902CB75926A80D07885BD9189"}
2024-04-25T08:02:05.260Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:02:08.822Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "7926C35BAACFCB20E6340AEFE16B9B699775A575AE76B2D206217C8A8C8A3068"}
2024-04-25T08:02:17.592Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:02:22.306Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "8B72747F91C2E47ED205CCEBFB49D14AA917F6F4B6272B6662104771D9D8E0AC"}
2024-04-25T08:02:28.325Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "1D17FFADCE03F7DB145ABE5CD4574BF7AB835279883B8FCA228CCF0BDEFDB663"}
2024-04-25T08:02:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460       Block cache might be too small. Metrics: hit: 2549995 miss: 64026217 keys-added: 6071197 keys-updated: 14219 keys-evicted: 6071175 cost-added: 62716680272679 cost-evicted: 62716440085157 sets-dropped: 191 sets-rejected: 50556719 gets-dropped: 4414464 gets-kept: 62158848 gets-total: 66576212 hit-ratio: 0.04
2024-04-25T08:02:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461       Cache life expectancy (in seconds):
 -- Histogram:
Min value: 0
Max value: 20
Count: 6070985
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6064710 99.90% 99.90%
[2, 4) 5064 0.08% 99.98%
[4, 8) 1113 0.02% 100.00%
[8, 16) 97 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
 --
2024-04-25T08:02:42.113Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:02:52.663Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:03:02.498Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "7FF1B0946DC8038981C5AC2E0089166D2F65CED4841E5AE753F55E6833C3AF79"}
2024-04-25T08:03:07.804Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "8025EC3F6A82D7BE5CADD1BFAADC713CA187B0712C9E1D29E7690700BDC0D8D9"}
2024-04-25T08:03:26.378Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "25375639C919C84FE9FF794F0763AFA1F94BA60BA6494C4EE5DE70EFA9644632"}
2024-04-25T08:03:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460       Block cache might be too small. Metrics: hit: 2557267 miss: 64187707 keys-added: 6086830 keys-updated: 14251 keys-evicted: 6086809 cost-added: 62890035729757 cost-evicted: 62889788131265 sets-dropped: 191 sets-rejected: 50702544 gets-dropped: 4414464 gets-kept: 62327616 gets-total: 66744974 hit-ratio: 0.04
2024-04-25T08:03:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461       Cache life expectancy (in seconds):
 -- Histogram:
Min value: 0
Max value: 20
Count: 6086619
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6080248 99.90% 99.90%
[2, 4) 5155 0.08% 99.98%
[4, 8) 1118 0.02% 100.00%
[8, 16) 97 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
 --
2024-04-25T08:04:10.705Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "CDC851B01241D59C13C1741105196A3F3C5D7A46941EC5F00B8BEC534ACE142A"}
2024-04-25T08:04:18.656Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "E3FBD06EB3D7A0AF5FE9166F8AAA8EAE7ED556105C0F84C18B4400DEA2534FCC"}
2024-04-25T08:04:18.819Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "78B586631F989A43DFD17808D2D2053256F86220D7057D7E98C5A274CDB73FEA"}
2024-04-25T08:04:22.114Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:04:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460       Block cache might be too small. Metrics: hit: 2565874 miss: 64377447 keys-added: 6104628 keys-updated: 14283 keys-evicted: 6104608 cost-added: 63074480913736 cost-evicted: 63074230159181 sets-dropped: 191 sets-rejected: 50874454 gets-dropped: 4414464 gets-kept: 62525888 gets-total: 66943321 hit-ratio: 0.04
2024-04-25T08:04:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461       Cache life expectancy (in seconds):
 -- Histogram:
Min value: 0
Max value: 20
Count: 6104418
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6097941 99.89% 99.89%
[2, 4) 5229 0.09% 99.98%
[4, 8) 1141 0.02% 100.00%
[8, 16) 106 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
 --
2024-04-25T08:04:42.662Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:04:52.383Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:04:54.566Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "B730DFD143FCCAAA729476FF88D5A185D0A1431042FD76F7C8CFC65DCB056345"}
2024-04-25T08:04:57.400Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:05:32.275Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "AFC302CAF4BE1F510F86F007C871082000E2F475BD5A5A82020307A859D8BD2C"}
2024-04-25T08:05:34.107Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:05:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460       Block cache might be too small. Metrics: hit: 2573486 miss: 64555066 keys-added: 6121690 keys-updated: 14304 keys-evicted: 6121664 cost-added: 63257915484247 cost-evicted: 63257658419777 sets-dropped: 191 sets-rejected: 51034987 gets-dropped: 4414464 gets-kept: 62711104 gets-total: 67128552 hit-ratio: 0.04
2024-04-25T08:05:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461       Cache life expectancy (in seconds):
 -- Histogram:
Min value: 0
Max value: 20
Count: 6121473
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6114915 99.89% 99.89%
[2, 4) 5299 0.09% 99.98%
[4, 8) 1151 0.02% 100.00%
[8, 16) 107 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
 --
2024-04-25T08:05:37.094Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "198675CD416AC9E8683078F684F31A9B7B03B8880449737A8F37D0340F0DE721"}
2024-04-25T08:05:41.729Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "208DE8C41F9BCAEEE03CFCB4DE52C08481F7F5F5DE213FC37EC226FEA294E28A"}
2024-04-25T08:05:42.113Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:05:51.581Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "E72FA5C69D5606399A45779A74E94F5CED7C760970D49BAD41EBE6D1955E4BA9"}
2024-04-25T08:05:56.242Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "1CEF49FC429D15905A56536D7A61E4F5E7BA0CE8064420A799BAD41496725393"}
2024-04-25T08:06:03.848Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "18EDB3A05FD7D380DE9B89B062DA96774AB3F79BF7450849DED0A639752DE049"}
2024-04-25T08:06:17.594Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:06:24.726Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "23CA4779BB5D99934E780BD1D392EBDB9DA2D2C193CCD9B52B7C85052E291C42"}
2024-04-25T08:06:29.646Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "E4DB13552DD58FFEE8D79A2B143CAE146F5AE24300F5C42AA1E4BEDD0A582EB3"}
2024-04-25T08:06:34.670Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "E0ECDA66217420CAACA75832CF88DF398BB103C06B48183876D8665A2E6AF0CC"}
2024-04-25T08:06:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460       Block cache might be too small. Metrics: hit: 2581764 miss: 64734414 keys-added: 6139809 keys-updated: 14340 keys-evicted: 6139778 cost-added: 63461873008696 cost-evicted: 63461611787744 sets-dropped: 191 sets-rejected: 51196183 gets-dropped: 4414592 gets-kept: 62898688 gets-total: 67316178 hit-ratio: 0.04
2024-04-25T08:06:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461       Cache life expectancy (in seconds):
 -- Histogram:
Min value: 0
Max value: 20
Count: 6139588
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6132951 99.89% 99.89%
[2, 4) 5359 0.09% 99.98%
[4, 8) 1167 0.02% 100.00%
[8, 16) 110 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
 --
2024-04-25T08:06:40.052Z        ERROR   shrex/eds       shrexeds/server.go:120  server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "ECE59C544DB0DBF97E07476A9DCE748E071811D17D22063677FF8DD3A4480DA5", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to acquire reader of mount so we can return the accessor: open /home/pops/.celestia-bridge-mocha-4/blocks/ECE59C544DB0DBF97E07476A9DCE748E071811D17D22063677FF8DD3A4480DA5: no such file or directory"}
2024-04-25T08:06:42.662Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:06:42.676Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "CC1A4A39102972A262B9838DE5ED522617C428F74B384221F7F0D14D2DBCE24A"}
2024-04-25T08:06:47.709Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "F2408B3EA85FB4180E61BD39F7A9B6AE1B54AAB5E932B383304E823B4349C321"}
2024-04-25T08:06:52.111Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:06:52.387Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "18CED3824302CF79512BA865F88D085243F8339325727ED5CB0CCB27F24A1683"}
2024-04-25T08:06:55.634Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "C45FDA45BD6D1E3224085B0818B21E664CAF83CA8D8F880228B8652E51A675C9"}
2024-04-25T08:06:58.821Z        ERROR   shrex/eds       shrexeds/server.go:120  server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "ABDAD379E096FF647535C92E223A87BE52523B1DE7BB9A5F5A15E84BDF20F0A0", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to acquire reader of mount so we can return the accessor: open /home/pops/.celestia-bridge-mocha-4/blocks/ABDAD379E096FF647535C92E223A87BE52523B1DE7BB9A5F5A15E84BDF20F0A0: too many open files"}
2024-04-25T08:07:01.315Z        ERROR   shrex/eds       shrexeds/server.go:120  server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "EBFA469A59788F4DEB110DABE40E5261E6D968AE198C5A494C9270B6D78EC901", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to acquire reader of mount so we can return the accessor: open /home/pops/.celestia-bridge-mocha-4/blocks/EBFA469A59788F4DEB110DABE40E5261E6D968AE198C5A494C9270B6D78EC901: too many open files"}
2024-04-25T08:07:03.688Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "E9F2D6E4B41AED0F80B5C9BFE45A40D24AB7AAA2276F5CD1EB98E07FFEFC2F25"}
2024-04-25T08:07:09.514Z        ERROR   shrex/eds       shrexeds/server.go:120  server: get CAR {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "3CD5329E3B82673F7393039FA4D0346CAF6B27667088DE453A541B49956A1597", "err": "failed to get accessor: failed to acquire shard: shard is in errored state; err: failed to recover index for shard 3CD5329E3B82673F7393039FA4D0346CAF6B27667088DE453A541B49956A1597: open /home/pops/.celestia-bridge-mocha-4/index/3CD5329E3B82673F7393039FA4D0346CAF6B27667088DE453A541B49956A1597.full.idx: too many open files"}
2024-04-25T08:07:12.829Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "E006FF1CF3DC10F9D60503A21E5942A59B7C4E1FD5844DCDCFB5E410A6EBAD82"}
2024-04-25T08:07:16.293Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "824573E032C9C68DFA0F7C27BDDD786368D41622C890F35CE40C56966AD478CD"}
2024-04-25T08:07:19.644Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "87A693BFC32DEB6E74398D673274E50F3B086D8AE1B5D9E1EAB2C45897C3AA27"}
2024-04-25T08:07:25.998Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "27CC927CEC0D24E51AD6B51F125D4660C73FB6F690D142C4D4E352E07230D6D0"}
2024-04-25T08:07:35.058Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460       Block cache might be too small. Metrics: hit: 2590472 miss: 64927948 keys-added: 6157559 keys-updated: 14374 keys-evicted: 6157535 cost-added: 63642968724664 cost-evicted: 63642704130515 sets-dropped: 191 sets-rejected: 51371933 gets-dropped: 4414848 gets-kept: 63100544 gets-total: 67518420 hit-ratio: 0.04
2024-04-25T08:07:35.058Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461       Cache life expectancy (in seconds):
 -- Histogram:
Min value: 0
Max value: 20
Count: 6157345
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6150587 99.89% 99.89%
[2, 4) 5454 0.09% 99.98%
[4, 8) 1190 0.02% 100.00%
[8, 16) 113 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
 --
2024-04-25T08:07:43.771Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "7C3A1ABDF12367045C17B0677574B4D20F3BAE593E734A6B65295B028D54F11C"}
2024-04-25T08:08:14.021Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "272979BAD3159480207DAC715156ED860FFF9132524BEA96EB6A29E90C26960A"}
2024-04-25T08:08:22.113Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:08:24.134Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "4B8B7BBC06C1DB078AB391EF19C68078F2B910E5292F21D01647958612D2F70B"}
2024-04-25T08:08:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460       Block cache might be too small. Metrics: hit: 2600072 miss: 65141854 keys-added: 6180385 keys-updated: 14410 keys-evicted: 6180362 cost-added: 63902059741804 cost-evicted: 63901800223991 sets-dropped: 191 sets-rejected: 51562947 gets-dropped: 4414912 gets-kept: 63324160 gets-total: 67741927 hit-ratio: 0.04
2024-04-25T08:08:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461       Cache life expectancy (in seconds):
 -- Histogram:
Min value: 0
Max value: 20
Count: 6180172
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6173381 99.89% 99.89%
[2, 4) 5486 0.09% 99.98%
[4, 8) 1191 0.02% 100.00%
[8, 16) 113 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
 --
2024-04-25T08:08:36.374Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWNEDvy2uYwBrtic5BzfDioKHDdSoTC72jwzHewNoVv1B3", "hash": "A630D138C083271F0878CEC1D3A00836CF70C932C8059DFCF1C88499089D120E"}
2024-04-25T08:08:42.589Z        INFO    rpc     rpc/server.go:108       server stopped
2024-04-25T08:08:42.589Z        INFO    header/p2p      p2p/server.go:86        server: stopping server
2024-04-25T08:08:42.662Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:08:48.708Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "79FAF1AE0EB44254CD311375C562E15642EC5D3862CAB52AB32BCB969A216A9B"}
2024-04-25T08:09:17.591Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:09:25.651Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "7DE740A56174AF919DC9BAB979A58DD6C056F1281AB94814A054495EC3AB9F18"}
2024-04-25T08:09:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460       Block cache might be too small. Metrics: hit: 2601525 miss: 65172940 keys-added: 6183944 keys-updated: 14415 keys-evicted: 6183927 cost-added: 63943646189944 cost-evicted: 63943381978736 sets-dropped: 191 sets-rejected: 51590499 gets-dropped: 4414912 gets-kept: 63356544 gets-total: 67774465 hit-ratio: 0.04
2024-04-25T08:09:35.057Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461       Cache life expectancy (in seconds):
 -- Histogram:
Min value: 0
Max value: 20
Count: 6183737
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 6176945 99.89% 99.89%
[2, 4) 5487 0.09% 99.98%
[4, 8) 1191 0.02% 100.00%
[8, 16) 113 0.00% 100.00%
[16, 32) 1 0.00% 100.00%
 --
2024-04-25T08:09:37.401Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWEEPSbGULYRURwpvSCJfYdjUn9hdtyoRo1NCVZkUMpWdY", "hash": "83FEECF5B531B3716DDA0219D2712566A22D5C6D4B5D9A8AF2AD1F9507756E40"}
2024-04-25T08:09:42.113Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:09:44.300Z        INFO    core    core/listener.go:203    listener: listening stopped
2024-04-25T08:09:44.302Z        WARN    shrex-sub       shrexsub/pubsub.go:83   unregistering topic     {"err": "no validator for topic mocha-4/eds-sub/v0.1.0"}
2024-04-25T08:09:44.303Z        INFO    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:546       Lifetime L0 stalled for: 0s
2024-04-25T08:09:44.367Z        INFO    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:625
Level 0 [ ]: NumTables: 01. Size: 13 MiB of 0 B. Score: 1.00->1.06 StaleData: 0 B Target FileSize: 64 MiB
Level 1 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 2 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 3 [B]: NumTables: 07. Size: 24 MiB of 25 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 4.0 MiB
Level 4 [ ]: NumTables: 46. Size: 253 MiB of 254 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 8.0 MiB
Level 5 [ ]: NumTables: 282. Size: 2.5 GiB of 2.5 GiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 16 MiB
Level 6 [ ]: NumTables: 1269. Size: 25 GiB of 25 GiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 32 MiB
Level 7 [ ]: NumTables: 5539. Size: 248 GiB of 248 GiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 64 MiB
Level Done
2024-04-25T08:09:45.910Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.910Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.910Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.910Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.910Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.910Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.910Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (local)
2024-04-25T08:09:45.911Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (local)
2024-04-25T08:09:45.911Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (local)
2024-04-25T08:09:45.911Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (local)
2024-04-25T08:09:45.911Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.911Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.912Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.912Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.912Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.912Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.912Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.912Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.912Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.913Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.914Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.915Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.916Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.916Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.916Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.916Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.916Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.916Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.916Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.916Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.916Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.917Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.917Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.917Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.917Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.917Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.917Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.917Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.917Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.917Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.918Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.919Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.919Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.919Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.919Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.919Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.919Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.919Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.919Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.919Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.920Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.920Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.920Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.920Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.920Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.920Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.920Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.920Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.920Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.921Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.921Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.921Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.921Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.921Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.921Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.921Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.921Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.922Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.922Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.923Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.923Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: stream reset
2024-04-25T08:09:45.926Z        INFO    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:546       Lifetime L0 stalled for: 0s
2024-04-25T08:09:46.085Z        INFO    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:625
Level 0 [ ]: NumTables: 00. Size: 0 B of 0 B. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 16 MiB
Level 1 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 2 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 3 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 4 [B]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB
Level 5 [ ]: NumTables: 16. Size: 55 MiB of 45 MiB. Score: 1.22->1.22 StaleData: 4.4 KiB Target FileSize: 4.0 MiB
Level 6 [ ]: NumTables: 90. Size: 454 MiB of 454 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 8.0 MiB
Level Done
2024-04-25T08:09:46.355Z        INFO    node    nodebuilder/module.go:26        Accessing keyring...
2024-04-25T08:09:46.382Z        INFO    module/state    state/keyring.go:40     constructed keyring signer      {"backend": "test", "path": "/home/pops/.celestia-bridge-mocha-4/keys", "key name": "my_celes_key", "chain-id": "mocha-4"}
2024-04-25T08:09:46.421Z        INFO    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/levels.go:171   All 106 tables opened in 24ms
2024-04-25T08:09:46.422Z        INFO    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/discard.go:66   Discard stats nextEmptySlot: 48
2024-04-25T08:09:46.422Z        INFO    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:368       Set nextTxnTs to 1799707427
2024-04-25T08:09:49.631Z        INFO    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/levels.go:118   6619 tables out of 7144 opened in 3s
2024-04-25T08:09:49.866Z        INFO    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/levels.go:171   All 7144 tables opened in 3.235s
2024-04-25T08:09:49.871Z        INFO    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/discard.go:66   Discard stats nextEmptySlot: 0
2024-04-25T08:09:49.874Z        INFO    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:368       Set nextTxnTs to 1071359
2024-04-25T08:09:49.878Z        INFO    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/value.go:592    Deleting empty file: /home/pops/.celestia-bridge-mocha-4/inverted_index//000371.vlog
2024-04-25T08:09:56.507Z        INFO    basichost       basic/natmgr.go:112     DiscoverNAT error:no NAT found
2024-04-25T08:09:59.874Z        INFO    share/discovery discovery/discovery.go:312      discovered wanted peers {"amount": 5}
2024-04-25T08:10:00.940Z        INFO    header/p2p      p2p/subscriber.go:81    joining topic   {"topic ID": "/mocha-4/header-sub/v0.0.1"}
2024-04-25T08:10:01.005Z        INFO    header/store    store/store.go:193      loaded head     {"height": 1685432, "hash": "D284D7162C514311161AFE9DD6CB6E55A7BE6BF901A88A36AB065DA2ABC23D7C"}
2024-04-25T08:10:01.005Z        INFO    fraudserv       fraudserv/helpers.go:29 joining topic   {"id": "/mocha-4/fraud-sub/badencodingv0.1/v0.0.1"}
2024-04-25T08:10:01.005Z        INFO    fraudserv       fraudserv/service.go:114        starting fraud proof service    {"protocol ID": "/mocha-4/fraud/v0.0.1"}
2024-04-25T08:10:01.446Z        INFO    header/sync     sync/sync_head.go:140   new network head        {"height": 1685527, "hash": "6EBE549FEBDD37258935D9106906E14A898ED695A48F463F404C439E8AC20369"}
2024-04-25T08:10:01.446Z        INFO    header/p2p      p2p/server.go:77        server: listening for inbound header requests   {"protocol ID": "/mocha-4/header-ex/v0.0.3"}
2024-04-25T08:10:01.447Z        INFO    header/sync     sync/sync.go:216        syncing headers {"from": 1685433, "to": 1685527}
2024-04-25T08:10:01.447Z        INFO    rpc     rpc/server.go:90        server started  {"listening on": "0.0.0.0:26658"}
2024-04-25T08:10:01.447Z        INFO    node    nodebuilder/node.go:113
/_____/  /_____/  /_____/  /_____/  /_____/
Started celestia DA node
node version:   v0.13.4
node type:      bridge
network:        mocha-4
/_____/  /_____/  /_____/  /_____/  /_____/
The p2p host is listening on:
*  /ip4/127.0.0.1/udp/2121/quic-v1/webtransport/certhash/uEiAktN9CODcWdqRyeVuGwDnt1Le97oOPipKvqbs0s-uTgQ/certhash/uEiC9AFyLT6belLezugelvCHt2GAQKg8fEs_fP5fVTvbzzg/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
*  /ip4/162.55.237.230/tcp/2121/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
*  /ip4/162.55.237.230/udp/2121/quic-v1/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
*  /ip4/162.55.237.230/udp/2121/quic-v1/webtransport/certhash/uEiAktN9CODcWdqRyeVuGwDnt1Le97oOPipKvqbs0s-uTgQ/certhash/uEiC9AFyLT6belLezugelvCHt2GAQKg8fEs_fP5fVTvbzzg/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
*  /ip6/::1/tcp/2121/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
*  /ip6/::1/udp/2121/quic-v1/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
*  /ip6/::1/udp/2121/quic-v1/webtransport/certhash/uEiAktN9CODcWdqRyeVuGwDnt1Le97oOPipKvqbs0s-uTgQ/certhash/uEiC9AFyLT6belLezugelvCHt2GAQKg8fEs_fP5fVTvbzzg/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
*  /ip6/2a01:4f8:271:3f6e::2/tcp/2121/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
*  /ip6/2a01:4f8:271:3f6e::2/udp/2121/quic-v1/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
*  /ip6/2a01:4f8:271:3f6e::2/udp/2121/quic-v1/webtransport/certhash/uEiAktN9CODcWdqRyeVuGwDnt1Le97oOPipKvqbs0s-uTgQ/certhash/uEiC9AFyLT6belLezugelvCHt2GAQKg8fEs_fP5fVTvbzzg/p2p/12D3KooWA92eW13sbg57xdXhrjfJTJf8t3frDy1bdvfCJLSmcD9F
2024-04-25T08:10:02.192Z        INFO    canonical-log   tls/transport.go:96     CANONICAL_PEER_STATUS: peer= addr=/ip4/34.145.72.169/tcp/19491 sample_rate=100 handshake_failure="tls" err="read tcp4 162.55.237.230:2121->34.145.72.169:19491: read: connection reset by peer"
2024-04-25T08:10:05.050Z        INFO    header/store    store/store.go:375      new head        {"height": 1685526, "hash": "AD7F3EA202A2E973DD44C48C0E33FB9ECCAB46AFEFADE6C371C831F3F7425500"}
2024-04-25T08:10:05.050Z        INFO    header/store    store/store.go:375      new head        {"height": 1685527, "hash": "6EBE549FEBDD37258935D9106906E14A898ED695A48F463F404C439E8AC20369"}
2024-04-25T08:10:05.051Z        INFO    header/sync     sync/sync.go:234        finished syncing headers        {"from": 1685433, "to": 1685527, "elapsed time": 3.604018927}
2024-04-25T08:10:06.555Z        INFO    canonical-log   swarm/swarm_listen.go:136       CANONICAL_PEER_STATUS: peer=12D3KooWKHePZ7CdFE9xgF7tHQxuPsj25DPFgWRuW3S7DK5gTtKk addr=/ip4/34.83.65.52/tcp/50168 sample_rate=100 connection_status="established" dir="inbound"
2024-04-25T08:10:09.447Z        INFO    header/store    store/store.go:375      new head        {"height": 1685528, "hash": "5B4E863EA638B9C750999C683F339B7A155AF66829140B589CA5E1A011560F22"}
2024-04-25T08:10:21.194Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "55020748E2B2C67A2C50F0DCD8AA72C8DA60541A422D63F1DA1439FC190C52BA"}
2024-04-25T08:10:21.475Z        INFO    header/store    store/store.go:375      new head        {"height": 1685529, "hash": "F1CBC8BAF54EF2CF2918D93B1ABCF6AACFF0F020E3DC381228A675E850AC750C"}
2024-04-25T08:10:23.317Z        INFO    canonical-log   swarm/swarm_listen.go:136       CANONICAL_PEER_STATUS: peer=12D3KooWSi9rYk3JF9iH9ZrvUJhu5YwiKynmjsomnrtUVomHL3kB addr=/ip4/34.145.72.169/tcp/18202 sample_rate=100 connection_status="established" dir="inbound"
2024-04-25T08:10:32.888Z        WARN    shrex/eds       shrexeds/server.go:116  server: request hash not found  {"peer": "12D3KooWR6SHsXPkkvhCRn6vp1RqSefgaT1X1nMNvrVjU2o3GoYy", "hash": "CA9F8B031E4490592ADAB06A484DF8462BC7A153B11D9439525C7E91ED233FE3"}
2024-04-25T08:10:33.073Z        INFO    header/store    store/store.go:375      new head        {"height": 1685530, "hash": "BD977AFDF2E8E3243193ACA1360EE6B1094E4661E2A6B81DCCCEFA4F22943D09"}
2024-04-25T08:10:44.959Z        INFO    header/store    store/store.go:375      new head        {"height": 1685531, "hash": "99E9A7601796525AAB4FDCD061FA23EBFC8D799EFE101383F8595E93A9D4BD91"}
2024-04-25T08:10:46.572Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:460       Block cache might be too small. Metrics: hit: 6170 miss: 927875 keys-added: 12014 keys-updated: 30 keys-evicted: 11999 cost-added: 117314086439 cost-evicted: 117061167309 sets-dropped: 0 sets-rejected: 123797 gets-dropped: 306624 gets-kept: 626496 gets-total: 934045 hit-ratio: 0.01
2024-04-25T08:10:46.572Z        WARN    badger4 v4@v4.2.1-0.20240106094458-1c417aa3799c/db.go:461       Cache life expectancy (in seconds):
 -- Histogram:
Min value: 0
Max value: 24
Count: 11994
50p: 2.00
75p: 2.00
90p: 2.00
[0, 2) 11913 99.32% 99.32%
[2, 4) 33 0.28% 99.60%
[4, 8) 24 0.20% 99.80%
[8, 16) 11 0.09% 99.89%
[16, 32) 13 0.11% 100.00%
 --
2024-04-25T08:10:51.185Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: timeout: no recent network activity
2024-04-25T08:10:52.112Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:10:56.505Z        INFO    header/store    store/store.go:375      new head        {"height": 1685532, "hash": "49A3AF2CAD3DB293A4A7A9949C674E2C0ADFB760628B49E25BC5FCC64E83D70A"}
2024-04-25T08:10:57.590Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:11:02.663Z        INFO    bitswap-server  server/server.go:547    Bitswap Client ReceiveError: Application error 0x0 (remote)
2024-04-25T08:11:08.184Z        INFO    header/store    store/store.go:375      new head        {"height": 1685533, "hash": "0682F70785C46981C78C6699B00766B134C6D45351175141D619D6DB30A3C6FD"}
2024-04-25T08:11:19.903Z        INFO    header/store    store/store.go:375      new head        {"height": 1685534, "hash": "FA98F3F2AB4EF47D2090A86F7130EFC5E124E7B7CD84BD032BA73881300E6F9C"}

@walldiss
Copy link
Member Author

Based on the logs, it is clear that resubscribing does not help resolve the issue, which I did not observe in other bridge nodes I checked earlier today.
@malise800 Thank you for providing the logs. Could you confirm if you are running multiple bridge nodes against the same RPC endpoint, and experiencing simultaneous lag across these nodes? Additionally, how often does the stall event occur?

@mindstyle85
Copy link

hey @walldiss we only run one, but we utilize our public rpc for it which is in the docs. But this hasnt been a problem at all since these latest versions (dont have exact timing but we didnt see this problem like a month ago or 2 months max). Also ive got another report with same problem and a private rpc used just for 1 bridge node.

As for often, its not that often, i think i know of about 4-5 incidents so far between mainnet and testnet

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@walldiss walldiss

Labels
area:core_and_app Relationship with Core node and Celestia-App bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@walldiss @mindstyle85 @malise800