my-dojo tracker stops adding blocks and the db container shows connection abortion error

[nickodev]

Hi, I have ran the dojo on the RPi and linux and and haven't experienced what I want to share in this issue. Some time ago I wanted to run the dojo on the windows OS to actually make it more inclusive. I will explain the process very shortly and will talk about the frustrating issue which I couldn't resolve yet.

After installing Docker desktop for windows, we can easily build the containers with the dojo.sh install command. It successfully built the containers and connected to my external bitcoind and by monitoring the node container's tracker output I could see the blocks were adding to the db container. But there is a problem there:
1- Upon installation the tracker will start and it adds about 7000~8000 blocks but suddenly it stops and after some time, the db container's log output will throw this error: "[Note] Aborted connection (number) to db: 'samourai-main" user: 'samourai host: '172.28.1.2' (Got an error reading communication packets)' and the tracker will start again with another connection. I mean when the connection 3 gets aborted, it will start with connection 4 for the next run.
(I know what happens internally, the wait-for mechanism will try to reconnect and start the tracker)
2- after the reconnect it will resume adding the block headers but this time it will stop sooner (after adding about 2000 blocks) and it waits for the reconnect.

I have spent a significant amount of time before issuing this bug here, and tried these solutions and they weren't effective:
1- I tried to increase max_packet_size and max_connections by adding them into the ~/my-dojo/mysql/mysql-dojo.cnf file.
2- I tried to create a delay on adding each block header.
3- Tried building it on another machine and the issue appeared exactly.

I will report on the progress of adding the block headers after 12 hours, but it will certainly be very very slow compared to RPi4 which takes around 4~5 hours.

Any help would be appretiated.

PS1: There is a ping every 30seconds in the mysql-db-wrapper and when the block addition stops (halts), the ping continues to run and each and every time it says there is 1 free connection

PS2: After restarting the docker service, it will start and add about 7~8K more blocks and then stops

edit: adding PS2

[nickodev]

I will report on the progress of adding the block headers after 12 hours

It got stock and couldn't finish the IBD for tracker.

[LaurentMT]

Can you provide the logs returned by these commands:

• ./dojo.sh logs tracker -n 1000
• ./dojo.sh logs tracker -d error -n 2000

[nickodev]

Sure,

./dojo.sh logs tracker -n 1000:

Error: Bitcoin Core JSON-RPC: host=x.x.x.x port=8332: Request error: connect ETIMEDOUT x.x.x.x:8332
    at ClientRequest. (/home/node/app/node_modules/bitcoind-rpc-client/lib/index.js:186:18)
    at emitOne (events.js:116:13)
    at ClientRequest.emit (events.js:211:7)
    at Socket.socketErrorListener (_http_client.js:387:9)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)
    at emitErrorNT (internal/streams/destroy.js:66:8)
    at _combinedTickCallback (internal/process/next_tick.js:139:11)
    at process._tickCallback (internal/process/next_tick.js:181:9)

./dojo.sh logs tracker -d error -n 2000:
One thing regarding this one. It starts initializing the tracker:

[20200226 09:58:44.911 054 MiB] Process ID: 192
[20200226 09:58:44.914 054 MiB] Preparing the tracker
[20200226 09:58:45.001 054 MiB] Created a database pool of 10 connections
[20200226 09:58:45.025 055 MiB] HTTP server listening on port 8082
[20200226 09:58:46.571 055 MiB] Tracker Startup (IBD mode)

Then it starts syncing the blocks but it stops after around adding 8000 blocks:

[20200226 09:58:46.603 059 MiB] Sync 619400 blocks
[20200226 09:58:47.677 287 MiB] Beginning to process new block header.
[20200226 09:58:47.688 287 MiB]  Added block header 1 (id=1)
...
...
[20200226 10:05:27.677 287 MiB] Beginning to process new block header.
[20200226 10:05:27.688 287 MiB]  Added block header 8105 (id=8105)

Again it tries to restart the tracker but this time less blocks are indexed and eventually it gets stuck, unless you restart the docker service itself.

One thing that I forgot to mention in my previous post is I get no warning/error in my bitcoind logs.

[LaurentMT]

Again it tries to restart the tracker but this time less blocks are indexed and eventually it gets stuck, unless you restart the docker service itself.

Your description suggests that the tracker is just waiting for new blocks dowloaded by bittcoind.

Next time the tracker seems stuck, can you check if bitcoind continues to download new blocks or if it seems stuck too?

[nickodev]

Next time the tracker seems stuck, can you check if bitcoind continues to download new blocks or if it seems stuck too?

I'm afraid the bitcoind is already at the tip.

[LaurentMT]

Can you share the complete log files of the tracker? (you can send them to my email address laurentmt145[at]gmail[dot]com).

They're 2 files named tracker-output.log and tracker-error.log and are located under the directory storing the docker volume of the nodejs container. Unfortunately, I don't know where this directory is on Windows.