Random ECONNRESET when talking to docker mysql

[TomYeoman]

Hi all,

I have a big issue where I randomly get ECONNRESET every 1-5 minutes when using mysql / mysql2 npm package. ( Works fine up until this point )

Environment

• mysql server running in docker, mapped to host port 3306
• node server running within docker, talking to mysql via host_ip:3306

Error

data-server-container |   Error: read ECONNRESET
data-server-container |
data-server-container |   - net.js:622 TCP.onread
data-server-container |     net.js:622:25

Fix attempts

I've tried all sorts to try and get the connection to remain - including

Using pools

const mysql = require('mysql');
const connection = mysql.createPool({
  connectionLimit: 10,
  host: process.env.DB_HOST_DOCKER,
  user: process.env.DB_USER_DOCKER,
  password: process.env.DB_PASSWORD_DOCKER,
  database: process.env.DB_DATABASE,
});

query = (sql) => {
  return new Promise(async (resolve, reject) => {
    connection.getConnection(function (err, poolConnection) {
      if (err) throw err; // not connected!

      // Use the connection
      poolConnection.query(sql, function (error, results, fields) {

        if (error) reject(error)

        resolve(results);

        // When done with the connection, release it.
        poolConnection.release();

      })

    })
  })
}

Creating a new connection on every request! ( Somehow this still gets ECONNRESET! )

query = (sql) => {

  const mysql = require('mysql');

  const connection = mysql.createConnection({
    host: process.env.DB_HOST_DOCKER,
    user: process.env.DB_USER_DOCKER,
    password: process.env.DB_PASSWORD_DOCKER,
    database: process.env.DB_DATABASE,
  });
  
  return new Promise(async (resolve, reject) => {
    connection.query(sql, (error, results) => {
      if (error) {
        reject(error);
      }
      resolve(results);
    });
  }

}

• Sending a ping to the connection every 10 seconds
• Checking my wait_timeout is high enough on mysql server ( 28800 as expected )

I'm running out of ideas here it's stopping me being able to use this in production :(

My best assumption after trying all the above is there an issue with the TCP Keep-Alive on the socket perhaps that is only triggered when running node / mysql within docker ( I stumbled on this MR - could this be a solution? #2110 )

[sqlbot]

mysql> SHOW STATUS LIKE 'uptime';

A problem causing the server daemon to crash and recover would be my first suspicion based on this report. Recovery from a crash can -- in some cases -- be quick enough to otherwise go unnoticed, but connections would be destroyed across such an event. This query fetches the current uptime for mysqld in seconds. If it resets to a lower value or is not as high as it seems like it should be based on how long the server is believed to have been continuously up, you likely have a server issue.

In any event, ECONNRESET is from either the TCP connection peer (the server, in this case, or a proxy/balancer, if you have one) or another intermediate network component and almost certainly something this module is only reporting on -- not contributing to.

[dougwilson]

That's a good thought to check @sqlbot ! @TomYeoman please let us know what you find from that. As far as some of the original post: (1) since both mysql and mysql2 are experiencing the exact same issue, it would seem to be from something they share in common, like the MySQL sever (as pointed out above), the Docker networking, or maybe something in Node.js itself and (2) I have never experienced this issue before, even when using Docker (which the CI for this module uses with success as well).

[TomYeoman]

Thanks for the suggestion @sqlbot , I just tested it out but it seems that i'm not experiencing any down time when running that command after my node server times out. This would make sense as i'm able to run a node server on localhost fine ( with mysql in docker ), but only when the node server is running within docker do I experience issues.

Perhaps if I post some of configuration it might help

docker-compose.js

services:
    data-server:
      container_name: bot-data-server-container
      image: bot-data-server
      build: ./data_server
      ports:
        - 5000:5000
    mysql:
      container_name: bot-db-container
      build: ./mysql_db
      environment:
        - MYSQL_ALLOW_EMPTY_PASSWORD=yes
      healthcheck:
          test: ["CMD", "mysqladmin" ,"ping", "-h", "localhost"]
          timeout: 20s
          retries: 10
      ports:
        - 3306:3306
      volumes:
      - ./mysql_db/my-data:/var/lib/mysql
      command: [
            '--wait_timeout=28800',
      ]

.env variables for the connection listed on original post ( Note how localhost works fine, but using the IP hangs up after 2-5min )- Seems likely there's a docker networking issue @dougwilson perhaps you can spot anything wrong?

# Docker - Must use hosts IPV4 address to talk to mysql running within docker on host
DB_HOST_DOCKER= X.X.X
DB_USER_DOCKER=root
DB_PASSWORD_DOCKER=xxx

# Development - Standard localhost can we used to talk to DB running in docker
DB_HOST_DEVELOPMENT=localhost
DB_USER_DEVELOPMENT=root
DB_PASSWORD_DEVELOPMENT=xxx

.Dockerfile for node server

FROM node:8-alpine

# Create app directory
WORKDIR /usr/src/app

# Install app dependencies
# A wildcard is used to ensure both package.json AND package-lock.json are copied
# where available (npm@5+)
COPY package*.json ./

RUN npm install
# If you are building your code for production
# RUN npm install --only=production

# Bundle app source
COPY . .

EXPOSE 5000
CMD ["yarn", "start:docker"]

.Dockerfile for mysql

FROM mysql:5.7.22
COPY init_db.sql /docker-entrypoint-initdb.d/

[dougwilson]

So your original post was about ECONNRESET. But now you're saying ECONNTIMEOUT which is a different error. Was that just a typo or are you getting both? Is it possible to enumerate each setup you tried and which error it was you got? That may help narrow down what is going on.

[TomYeoman]

We can close this issue, I was being stupid it was another library throwing the ECONNRESET ( ended up being puppeteer with a terrible stackdump ) 😠 .

Thanks for the help anyway!

[shyamjith94]

HI. I am facing the same issue, please help thanks