Elasticsearch: Don't throw exception on missing CA cert file

[spinscale]

The current check throws an exception if the version is supposed to be
greater than 8 and the ca cert file is missing. This posed two problems:

1. A custom docker image might have a 'latest' version, and thus is
   newer than version 8
2. A user may have configured their own CA in a different path

In order to accomodate for this, a missing copy of a the ca crt file
will log a warning, but not throw an exception and leave the container
unstarted. There is also another public method, allowing to read the
cert, called extractCert.

Closes #5264

[kiview]

Thanks for the prompt investigation and fix @spinscale, much appreciated 🙌
I added a comment regarding the API.

[kiview]

LGTM, thanks!
Only thing I was wondering if we should maybe also add a test utilizing withCertPath if it is not too involved.

[spinscale]

Hm, I could copy another http_ca.crt into the container, but that would need to be done before Elasticsearch starts, is this possible? container.copyFileToContainer requires a running container IIRC

[kiview]

withCopyFileToContainer will perform the copying on startup 🙂

[spinscale]

oooh... of course. thx. I took a small shortcut and added a http_ca.crt file, that will not match the one created by Elasticsearch on startup and therefore fails on purpose when trying to connect to Elasticsearch using that for a client with a handshake exception.

[bsideup]

could you please add a test that verifies that #5264 works correctly now?

[spinscale]

done, tagged an older image as latest and started the container. If there is a better way, let me know, I stole a little code from DockerComposeLocalImageTest

[bsideup]

💯

[kiview]

Merged, thanks a lot for this high-quality PR @spinscale, a pleasure to work with you 🥳

[vmassol]

Hey guys (cc @spinscale), I'm using tag 8.2.0for ES and v1.17.1 for TC and I'm getting this exception:

13:16:56.855 [main] ERROR ?.e.c.2.0] - Could not start container
com.github.dockerjava.api.exception.NotFoundException: Status 404: {"message":"Could not find the file /usr/share/elasticsearch/config/certs/http_ca.crt in container 8edfaa2eef41d2bc12dae07291f3796973ce3670440d918fb7ec90a6a5410a96"}

I understand that this PR (when released), will not throw this exception anymore but that ES will still not start.

I'm starting it with the following code:

        DockerImageName imageName =
            DockerImageName.parse("docker.elastic.co/elasticsearch/elasticsearch").withTag("8.2.0");
        container = new ElasticsearchContainer(imageName);
        // - single node since we don't need a cluster for testing
        // - disabled security to avoid the warning messages and because security is not needed for testing as
        //   the data is discarded and the testing environment is secure.
        container.setEnv(List.of(
            "discovery.type=single-node",
            "xpack.security.enabled=false",
            "xpack.security.http.ssl.enabled=false",
            "xpack.security.transport.ssl.enabled=false"));
        container.start();

As you can see, I've tried to turn off security so I'm not sure why it doesn't work in 8.2.0. It works fine in 7.17.3.

Also, I'd be interested to understand how I can start ES 8.2.0 with TC in a secure mode. I've tried to read https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html but I'm not sure I understand everything. Am I supposed to provide the http_ca.crt file or is generated by the docker image? Step 4 suggests it's generated. So how are we supposed to use ES with TC? Do we have to start the container once, copy the crt file on the host, stop the container, restart it again with a volume mapping on the host for the cert file? Sounds pretty complex :)

Thx!

[spinscale]

Can you try #5264 (comment) as a workaround until the next version is released?

[vmassol]

@spinscale that works fine, thx!

[vmassol]

I confirm it's now working OOB with TC 1.17.2. Thanks for that!

However, I still get a warning:

10:38:15.887 [main] WARN  o.t.e.ElasticsearchContainer - CA cert under /usr/share/elasticsearch/config/certs/http_ca.crt not found.

I have turned security off and thus wasn't expecting any warning. Is that something that could be improved? Should I create an issue for it @spinscale ?

       DockerImageName imageName =
           DockerImageName.parse("docker.elastic.co/elasticsearch/elasticsearch").withTag(ELASTICSEARCH_VERSION);
       ElasticsearchContainer container = new ElasticsearchContainer(imageName);
       // - single node since we don't need a cluster for testing
       // - disabled security to avoid the warning messages and because security is not needed for testing as
       //   the data is discarded and the testing environment is secure.
       container.setEnv(List.of("discovery.type=single-node", "xpack.security.enabled=false"));
       container.start();

Thx