in-cluster: Name does not resolve

[JPFrancoia]

Current and expected behavior

When running in-cluster, it looks like my client can't resolve the name of the api. Locally, everything runs fine but here are the logs when I'm running in-cluster:

[2022-11-03T17:27:50Z TRACE kube_client::config] no local config found, falling back to local in-cluster config error=failed to read kubeconfig from '"/home/raftcat/.kube/config"': No such file or directory (os error 2)
...
[2022-11-03T17:27:50Z DEBUG kube_client::client::builder] HTTP; http.method=GET http.url=https://kubernetes.default.svc/apis/babylontech.co.uk/v1/namespaces/dev/shipcatmanifests? otel.name="list" otel.kind="client"
[2022-11-03T17:27:50Z DEBUG kube_client::client::builder] requesting
[2022-11-03T17:27:50Z DEBUG kube_client::client::builder] HTTP; http.method=GET http.url=https://kubernetes.default.svc/apis/babylontech.co.uk/v1/namespaces/dev/shipcatconfigs? otel.name="list" otel.kind="client"
[2022-11-03T17:27:50Z DEBUG kube_client::client::builder] requesting
[2022-11-03T17:27:50Z DEBUG kube_client::client::builder] HTTP; otel.status_code="ERROR"
[2022-11-03T17:27:50Z ERROR kube_client::client::builder] failed with error error trying to connect: dns error: failed to lookup address information: Name does not resolve
[2022-11-03T17:27:50Z ERROR raftcat::state] Kubernetes stream failure: failed to perform initial object list: HyperError: error trying to connect: dns error: failed to lookup address information: Name does not resolve
thread 'main' panicked at 'Kubernetes stream exited', /volume/raftcat/src/state.rs:97:25
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[2022-11-03T17:27:50Z DEBUG kube_client::client::builder] HTTP; otel.status_code="ERROR"
[2022-11-03T17:27:50Z ERROR kube_client::client::builder] failed with error error trying to connect: dns error: failed to lookup address information: Name does not resolve
[2022-11-03T17:27:50Z ERROR raftcat::state] Kubernetes stream failure: failed to perform initial object list: HyperError: error trying to connect: dns error: failed to lookup address information: Name does not resolve
thread 'main' panicked at 'Kubernetes stream exited', /volume/raftcat/src/state.rs:116:25

I did see these issues:

• dns lookup error to kubernetes.default.svc in-cluster when using hostNetwork #953
• use cluster dns to reach apiserver for rustls - for #587 #597
• In-cluster config uses an IP address instead of recommended DNS name #587

But none of the workarounds or solutions mentioned there could help solve my problem.

What I have tried:

I tried patching the deployment of my service with this:

+      hostNetwork: true
+      dnsPolicy: ClusterFirstWithHostNet

Like it's mentioned there: #953 (comment)

I also tried setting the env var KUBERNETES_SERVICE_HOST = kubernetes.default.svc like it's mentioned here: #587 (comment), even though it was a bit useless since this workaround was merged.

I also tried to build and run without rustls-tls, but then I got different error messages (EDIT: see below for why this happens. I probably jumped the gun, it was probably working):

[2022-11-03T17:42:22Z TRACE kube_client::config] no local config found, falling back to local in-cluster config error=failed to read kubeconfig from '"/home/raftcat/.kube/config"': No such file or directory (os error 2)
...
[2022-11-03T17:42:22Z DEBUG kube_client::client::builder] HTTP; http.method=GET http.url=https://172.20.0.1/apis/babylontech.co.uk/v1/namespaces/dev/shipcatmanifests? otel.name="list" otel.kind="client"
[2022-11-03T17:42:22Z DEBUG kube_client::client::builder] requesting
[2022-11-03T17:42:22Z DEBUG kube_client::client::builder] HTTP; http.method=GET http.url=https://172.20.0.1/apis/babylontech.co.uk/v1/namespaces/dev/shipcatconfigs? otel.name="list" otel.kind="client"
[2022-11-03T17:42:22Z DEBUG kube_client::client::builder] requesting
[2022-11-03T17:42:22Z DEBUG kube_client::client::builder] HTTP; otel.status_code="ERROR"
[2022-11-03T17:42:22Z ERROR kube_client::client::builder] failed with error error trying to connect: tcp connect error: Connection refused (os error 111)
[2022-11-03T17:42:22Z ERROR raftcat::state] Kubernetes stream failure: failed to perform initial object list: HyperError: error trying to connect: tcp connect error: Connection refused (os error 111)
thread 'main' panicked at 'Kubernetes stream exited', /volume/raftcat/src/state.rs:97:25
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[2022-11-03T17:42:22Z DEBUG kube_client::client::builder] HTTP; otel.status_code="ERROR"
[2022-11-03T17:42:22Z ERROR kube_client::client::builder] failed with error error trying to connect: tcp connect error: Connection refused (os error 111)
[2022-11-03T17:42:22Z ERROR raftcat::state] Kubernetes stream failure: failed to perform initial object list: HyperError: error trying to connect: tcp connect error: Connection refused (os error 111)
thread 'main' panicked at 'Kubernetes stream exited', /volume/raftcat/src/state.rs:116:25

Would you have any idea about what's going on?

Possible solution

No response

Additional context

No response

Environment

Client Version: v1.21.14
Server Version: v1.21.14-eks-6d3986b

Configuration and features

kube = { version = "0.76.0", features=["rustls-tls", "runtime", "client"], default-features = false}
k8s-openapi = { version = "0.16.0", features = ["v1_25"] }
reqwest = { version = "0.11.12" , default-features = false, features=["rustls-tls", "json"]}

Affected crates

No response

Would you like to work on fixing this bug?

No response

[clux]

This is a combination of #153 + the fact that the dns workaround we had in place is no longer default (because it's not a recommended Kubernetes way to do things). See #1003

Using the Config::incluster_dns ctor should fix it. As should changing the tls stacks to the default openssl.

[JPFrancoia]

I tried with:

    let cfg = if let Ok(c) =
        kube::config::Config::from_kubeconfig(&kube::config::KubeConfigOptions::default()).await
    {
        c
    } else {
        kube::config::Config::incluster_dns().expect("Failed to load kube config")
    };

But it didn't work (same error).

However running with the openssl stack seems to work:

kube = { version = "0.76.0", features=["openssl-tls", "runtime", "client"], default-features = false}
k8s-openapi = { version = "0.16.0", features = ["v1_25"] }

BUT, weirdly, when I check the logs I get this first:

[2022-11-04T12:00:17Z DEBUG kube_client::client::builder] HTTP; http.method=GET http.url=https://172.20.0.1/apis/babylontech.co.uk/v1/namespaces/dev/shipcatmanifests? otel.name="list" otel.kind="client"
[2022-11-04T12:00:17Z DEBUG kube_client::client::builder] requesting
[2022-11-04T12:00:17Z DEBUG kube_client::client::builder] HTTP; http.method=GET http.url=https://172.20.0.1/apis/babylontech.co.uk/v1/namespaces/dev/shipcatconfigs? otel.name="list" otel.kind="client"
[2022-11-04T12:00:17Z DEBUG kube_client::client::builder] requesting
[2022-11-04T12:00:17Z DEBUG kube_client::client::builder] HTTP; otel.status_code="ERROR"
[2022-11-04T12:00:17Z ERROR kube_client::client::builder] failed with error error trying to connect: tcp connect error: Connection refused (os error 111)

There seems to be a panic, but shortly after it resumes and I get proper access. Weird.

Anyway, it looks like it works now!

[clux]

You on istio? Rust apps tend to boot faster than istio so the first connection attempt tends to fail until the sidecar has established the connection routing.

[JPFrancoia]

You on istio

Yes. That must be it then!

Thanks again for your help.