Temporal can't connect to Elasticsearch with enabled TLS

[ganievs]

Expected Behavior

Temporal cluster establishes connection with Elasticsearch.

Actual Behavior

"no ElasticSearch node available"

Steps to Reproduce the Problem

1. Setup elastic 8 version with enabled TLS on the HTTP layer
2. Set scheme: "https" in Temporal server config

        elasticsearch:
          version: "v8"
          url:
            scheme: "https"

Specifications

• Temporal Version: 1.21.3
• Elasticsearch Version: 8.6.2
• Platform: Kubernetes

[ganievs]

Hi all! Seems like the client library that you're using, doesn't have actual support for 8 version of Elasticsearch and not maintained anymore olivere/elastic#1533.
Do you have any plans to move to the official lib?

[rodrigozhou]

Did you set up the certs correctly? Are they from a known issuer?
Please provide more details of how you tried set up.

[ganievs]

I do requests to the Elasticsearch API from my host with the same credentials and everything works.
Also, I use https://github.com/prometheus-community/elasticsearch_exporter which connects to the Elasticsearch instance and it also works well. Based on this, I can say that the certificates are configured correctly.

Are they from a known issuer?

Yes, I used Let's Encrypt for certificate request

[rodrigozhou]

Can you share your config? Does it look like this?

      elasticsearch:
        version: "v8"
        url:
          scheme: "https"
          host: "127.0.0.1:9200"
        username: <username>
        password: <password>

[ganievs]

Can you share your config?

Unfortanly I destroyed a test environment with the setup. If it would help, I'll spin up the env later and give you debug info.

Does it look like this?

Yes, it is.

[rodrigozhou]

I followed these steps to setup ES with certs and authentication, and it worked.

In those instructions, it uses the ES tool to generate certs which are not from a known issuer, so I had to set a custom http client to Temporal Server that skip this check. Something like this:

temporal.NewServer(
  temporal.WithElasticsearchHttpClient(
    &http.Client{
      Transport: &http.Transport{
        TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
      },
    },
  ),
)

[ganievs]

I also confirm that works! Would be helpful to mention the ability to define a custom HTTP client for Elasticsearch in server-opritons on docs.temporal.io