Random ECONNRESET when talking to docker mysql

[TomYeoman]

Hi all,

I have a big issue where I randomly get ECONNRESET every 1-5 minutes when using mysql / mysql2 npm package. ( Works fine up until this point )

Environment

• mysql server running in docker, mapped to host port 3306
• node server running within docker, talking to mysql via host_ip:3306

Errot

data-server-container |   Error: read ECONNRESET
data-server-container |
data-server-container |   - net.js:622 TCP.onread
data-server-container |     net.js:622:25

Fix attempts

I've tried all sorts to try and get the connection to remain - including

Using pools

const mysql = require('mysql2');
const connection = await mysql.createPool({
  connectionLimit: 10,
  host: process.env.DB_HOST_DOCKER,
  user: process.env.DB_USER_DOCKER,
  password: process.env.DB_PASSWORD_DOCKER,
  database: process.env.DB_DATABASE,
});

query = (sql) => {
  return new Promise(async (resolve, reject) => {
    connection.getConnection(function (err, poolConnection) {
      if (err) throw err; // not connected!

      // Use the connection
      poolConnection.query(sql, function (error, results, fields) {

        if (error) reject(error)

        resolve(results);

        // When done with the connection, release it.
        poolConnection.release();

      })

    })
  })
}

Creating a new connection on every request! ( Somehow this still gets ECONNRESET! )

query = (sql) => {

  const mysql = require('mysql2');

  const connection = await mysql.createConnection({
    host: process.env.DB_HOST_DOCKER,
    user: process.env.DB_USER_DOCKER,
    password: process.env.DB_PASSWORD_DOCKER,
    database: process.env.DB_DATABASE,
  });
  
  return new Promise(async (resolve, reject) => {
    connection.query(sql, (error, results) => {
      if (error) {
        reject(error);
      }
      resolve(results);
    });
  }

}

• Sending a ping to the connection every 10 seconds
• Checking my wait_timeout is high enough on mysql server ( 28800 as expected )

I'm running out of ideas here it's stopping me being able to use this in production :(

My best assumption after trying all the above is there an issue with the TCP Keep-Alive on the socket perhaps that is only triggered when running node / mysql within docker ( I stumbled on this MR - could this be a solution? mysqljs/mysql#2110 )

[eagle7410]

I have the same problems.
Docker swarm
mysql/server 5.7
node 8.11
mysql2: 1.6.1

[hassan-jahan]

Same issue here :|

[TomYeoman]

Just a heads up I was actually receiving it from another part of my application ( unrelated to MySql in the end - I believe it was puppeteer playing up in a docker environment ) so just make sure you've completely eliminated that as a case!

[paramsinghvc]

Facing the same sort of problem in my docker setup

Error: read ECONNRESET
at TCP.onread

[sidorares]

@paramsinghvc is there a reliable way to reproduce?

[hassan-jahan]

The problem is related to docker swarm and pooling.

Disable pooling (in knex file : pool: {min:0} )

Or check out this:
https://success.docker.com/article/ipvs-connection-timeout-issue

[erming]

I had the same problem until I added command: "--wait_timeout=28800" to my docker-compose file. I'm running MariaDB though, but it might be the same problem.

Example docker-compose:

version: "3.7"
services:
  mysql:
    image: mariadb
    ports:
      - 3306:3306
    volumes:
      - mysql-volume:/var/lib/mysql
    environment:
      MYSQL_ROOT_PASSWORD: foo
      MYSQL_DATABASE: bar
    restart: always
    command: "--wait_timeout=28800"
volumes:
  mysql-volume:

Here's a link to the MariaDB documentation:
https://mariadb.com/kb/en/server-system-variables/#wait_timeout

[whtjs]

If somebody still has this issue, try to update docker. In my case update solves the problem

[benyaminl]

If somebody still has this issue, try to update docker. In my case update solves the problem

Do you use Docker Swarm?
TBFrank I'm quite hitting the wall with this and tried to check the IPVS on the docker swarm, because it seems the connection is keep opened via mysql session manager, but IPVS already drop the request that come in from typeORM that use mysql2 pools... :/

[whtjs]

If somebody still has this issue, try to update docker. In my case update solves the problem

Do you use Docker Swarm?
TBFrank I'm quite hitting the wall with this and tried to check the IPVS on the docker swarm, because it seems the connection is keep opened via mysql session manager, but IPVS already drop the request that come in from typeORM that use mysql2 pools... :/

Nope, I was not using docker swarm