Race condition when starting container causes IndexOutOfBoundsException when binding ports

[ewencluley]

Description

An intermittent issue can be seen when starting containers. It seems to be a race condition that is exacerbated when running maven with debug logging -X. Without the debug logging containers fail to start with

[ERROR] DOCKER> Error occurred during container startup, shutting down...

The full stack trace of the exception is:

Caused by: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
    at jdk.internal.util.Preconditions.outOfBounds (Preconditions.java:64)
    at jdk.internal.util.Preconditions.outOfBoundsCheckIndex (Preconditions.java:70)
    at jdk.internal.util.Preconditions.checkIndex (Preconditions.java:248)
    at java.util.Objects.checkIndex (Objects.java:372)
    at java.util.ArrayList.get (ArrayList.java:459)
    at com.google.gson.JsonArray.get (JsonArray.java:194)
    at io.fabric8.maven.docker.model.ContainerDetails.createPortBindings (ContainerDetails.java:200)
    at io.fabric8.maven.docker.model.ContainerDetails.getPortBindings (ContainerDetails.java:139)
    at io.fabric8.maven.docker.service.RunService.updateMappedPortsAndAddresses (RunService.java:461)
    at io.fabric8.maven.docker.service.RunService.createAndStartContainer (RunService.java:156)
    at io.fabric8.maven.docker.service.helper.StartContainerExecutor.startContainer (StartContainerExecutor.java:47)
    at io.fabric8.maven.docker.StartMojo.lambda$startImage$0 (StartMojo.java:299)

When adding some logging to the maven plugin I can see that during the the io.fabric8.maven.docker.model.ContainerDetails.createPortBindings method call the ports will usually have some mappings like this:

Ports:{"5432/tcp":[{"HostIp":"0.0.0.0","HostPort":"54290"}],}

However sometimes the array of mapped ports will be empty, e.g.

Ports:{"5432/tcp":[]}

This causes the exception as the array is expected to not be empty and the first element retrieved.

I believe the call to the api in io.fabric8.maven.docker.service.RunService.updateMappedPortsAndAddresses is sometimes returning this empty data for the port if called too early.

The solution I would propose would be to retry the API call and the createPortBindings activity if an empty array is encountered.

Info

• docker-maven-plugin version : 0.34.1
• Maven version (mvn -v) :

Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /usr/local/Cellar/maven/3.6.3_1/libexec
Java version: 11.0.10, vendor: AdoptOpenJDK, runtime: /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home
Default locale: en_GB, platform encoding: UTF-8
OS name: "mac os x", version: "10.14.6", arch: "x86_64", family: "mac"

• Docker version : 3.2.2
• If it's a bug, how to reproduce : The issue is intermittent so impossible to consistently reproduce. It appeared to be more easily reproducible when running mvn verify with -X flag.

[ewencluley]

Opened PR with retry logic to address, happy to receive and implement feedback on it.
#1447

[git9999999]

hi

the code in the pr looks good to me, with the retry. but unfortunate it does not work correct :-(

Failed to execute goal io.fabric8:docker-maven-plugin:0.34-SNAPSHOT-FIX:start (start-docker-ms-backend) on project ms-backend-application: Execution start-docker-ms-backend of goal io.fabric8:docker-maven-plugin:0.34-SNAPSHOT-FIX:start failed: Index 0 out of bounds for length 0

i am happy to run it with more log information if you need.

[rohanKanojia]

@git9999999 : Would appreciate if you could provide logs with -X enabled

[ewencluley]

Hmm, odd. Our stack traces are identical so im unsure why it's not working. I would suggest running with the -X flag and see if you can see the Failed to get port bindings for container log statement.

Also, know this might sound stupid but if you have been locally installing (mvn install) your own version of the plugin it might be worth clearing out the local .m2 repo or at least that artifact from it. Ive had instances in the past where maven has got a little confused about snapshot versions.

[git9999999]

hi

here we go. i checked out the PR branch and changed the name of the version to 0.34-SNAPSHOT-FIX. the build it. i also updated this version in my projekt. so i am sure this is the right version of the plugin.
i hope you see what you need. if not, you can add additional logs and i can run the test again.

mvn_clean_install_with_fixed_plugin_from_pr_8f0f0093.txt

cheers

[ewencluley]

So im not seeing that log message and the line indicated by the stack trace io.fabric8.maven.docker.model.ContainerDetails.createPortBindings (ContainerDetails.java:200) is (in my code)

} else {

which cant be throwing that exception.
So it really looks like the updated code is not being used.

[rohanKanojia]

@git9999999: Do you have some reproducer project and steps we can try out to reproduce this issue? It can help @ewencluley to debug what's not working as expected.

[ewencluley]

Alternatively here's a built version of my code. Pop that in your .m2/repository/io/fabric8/docker-maven-plugin directory and use the version 0.34-EWENCLULEY.

0.34-EWENCLULEY.zip

But also 100% happy to take a look at an example project if you can provide as suggested ^

[git9999999]

Good Morning :-)
now i did the test right, and now it works. :-)
thanks for fixing the plugin.
cheers g