Synchronize removal of containers from JDBC URL cache

[edee111]

Let's say test1 is SpringBootTest marked with @DirtiesContext -> after execution the whole Spring Context is thrown away -> DataSource is closed with all its connections.

This scenario may happen (real example):

• test1 starts
• test1 ends
• test2 starts
• test2 creates new data source within spring context - ContainerDatabaseDriver::connect has passed the check in the cache -> container is there -> no need to create a new container
• test1 context is thrown away -> test1 data source connections are being closed -> ContainerDatabaseDriver::wrapConnection::ConnectionWrapper on the last connection stops the container
• test2 throws an exception ->
  Caused by: java.sql.SQLException: Could not create new connection
  at org.testcontainers.containers.JdbcDatabaseContainer.createConnection(JdbcDatabaseContainer.java:228) ~[jdbc-1.16.1.jar:na]

In my case, approximately 1 in 50 pipeline builds fails with the exception. Here is also part of the log that clearly shows in logs the situation described above:

Test1 - container init:

12:08:32.715 [Test worker 13] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Starting...
12:08:32.870 [Test worker 13] DEBUG o.t.jdbc.ContainerDatabaseDriver - Container not found in cache, creating new instance
12:08:32.925 [Test worker 13] DEBUG o.t.u.TestcontainersConfiguration - Testcontainers configuration overrides will be loaded from file:/root/.testcontainers.properties
12:08:32.938 [Test worker 13] WARN  o.t.u.TestcontainersConfiguration - Attempted to read Testcontainers configuration file at file:/root/.testcontainers.properties but the file was not found. Exception message: FileNotFoundException: /root/.testcontainers.properties (No such file or directory)
12:08:32.948 [Test worker 13] DEBUG o.t.d.DockerClientProviderStrategy - Trying out strategy: EnvironmentAndSystemPropertyClientProviderStrategy
12:08:33.273 [ducttape-0 23] DEBUG o.t.d.DockerClientProviderStrategy - Pinging docker daemon...
12:08:33.304 [ducttape-0 23] DEBUG o.t.s.c.g.d.c.command.AbstrDockerCmd - Cmd: 
12:08:33.618 [Test worker 13] INFO  o.t.d.DockerClientProviderStrategy - Found Docker environment with Environment variables, system properties and defaults. Resolved dockerHost=tcp://docker:2375
12:08:33.618 [Test worker 13] DEBUG o.t.d.DockerClientProviderStrategy - Transport type: 'httpclient5', Docker host: 'tcp://docker:2375'
12:08:33.618 [Test worker 13] DEBUG o.t.d.DockerClientProviderStrategy - Checking Docker OS type for Environment variables, system properties and defaults. Resolved dockerHost=tcp://docker:2375
12:08:33.619 [Test worker 13] INFO  o.testcontainers.DockerClientFactory - Docker host IP address is docker
12:08:33.620 [Test worker 13] DEBUG o.t.s.c.g.d.c.command.AbstrDockerCmd - Cmd: 
12:08:33.652 [Test worker 13] DEBUG o.t.s.c.g.d.c.command.AbstrDockerCmd - Cmd: 
12:08:33.653 [Test worker 13] TRACE o.t.s.c.g.d.core.exec.VersionCmdExec - GET: DefaultWebTarget{path=[/version], queryParams={}}
12:08:33.685 [Test worker 13] INFO  o.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 19.03.14
  API Version: 1.40
  Operating System: Alpine Linux v3.12 (containerized)
  Total Memory: 31993 MB

then test2 spring context starting:

12:09:37.101 [Test worker 13] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-2 - Starting...
12:09:37.102 [Test worker 13] DEBUG o.t.s.c.g.d.c.command.AbstrDockerCmd - Cmd: af5ed568d66f93bc6932ce9126c83f427d0baedd5c5d724ebc523857f9942aa3,false
12:09:37.102 [Test worker 13] DEBUG o.t.s.c.g.d.c.e.InspectContainerCmdExec - GET: DefaultWebTarget{path=[/containers/af5ed568d66f93bc6932ce9126c83f427d0baedd5c5d724ebc523857f9942aa3/json], queryParams={}}
12:09:42.205 [HikariPool-1 connection closer 29] TRACE o.t.utility.ResourceReaper - Stopped container: mariadb:10.5.9
12:09:42.207 [HikariPool-1 connection closer 29] DEBUG o.t.s.c.g.d.c.command.AbstrDockerCmd - Cmd: af5ed568d66f93bc6932ce9126c83f427d0baedd5c5d724ebc523857f9942aa3,false
12:09:42.208 [HikariPool-1 connection closer 29] DEBUG o.t.s.c.g.d.c.e.InspectContainerCmdExec - GET: DefaultWebTarget{path=[/containers/af5ed568d66f93bc6932ce9126c83f427d0baedd5c5d724ebc523857f9942aa3/json], queryParams={}}
12:09:42.210 [HikariPool-1 connection closer 29] TRACE o.t.utility.ResourceReaper - Removing container: af5ed568d66f93bc6932ce9126c83f427d0baedd5c5d724ebc523857f9942aa3
12:09:42.211 [HikariPool-1 connection closer 29] DEBUG o.t.s.c.g.d.c.command.AbstrDockerCmd - Cmd: af5ed568d66f93bc6932ce9126c83f427d0baedd5c5d724ebc523857f9942aa3,true,true
12:09:42.212 [HikariPool-1 connection closer 29] TRACE o.t.s.c.g.d.c.e.RemoveContainerCmdExec - DELETE: DefaultWebTarget{path=[/containers/af5ed568d66f93bc6932ce9126c83f427d0baedd5c5d724ebc523857f9942aa3], queryParams={v=[true], force=[true]}}
12:09:42.353 [HikariPool-1 connection closer 29] DEBUG o.t.utility.ResourceReaper - Removed container and associated volume(s): mariadb:10.5.9
12:09:43.214 [Test worker 13] ERROR com.zaxxer.hikari.pool.HikariPool - HikariPool-2 - Exception during pool initialization.
java.sql.SQLException: Could not create new connection
	at org.testcontainers.containers.JdbcDatabaseContainer.createConnection(JdbcDatabaseContainer.java:228)
	at org.testcontainers.jdbc.ContainerDatabaseDriver.connect(ContainerDatabaseDriver.java:124)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138)
	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:364)
	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206)