etcd is unreachable, please check your endpoints #12877

DougTrajano · 2021-08-04T20:55:38Z

Error: Unable to decrypt an encrypted ETCD backend for IAM users and policies: etcd is unreachable

Expected Behavior

I expected to use etcd to store IAM users and policies as I'm using minIO Gateway. My goal is to setup minIO STS (with Keycloak).

Current Behavior

I'm getting an error message that says that etcd is unreachable.

API: SYSTEM()
Time: 20:22:51 UTC 08/04/2021
DeploymentID: d28fda90-0399-453a-9fe3-fc83921ef50c
Error: Unable to decrypt an encrypted ETCD backend for IAM users and policies: etcd is unreachable, please check your endpoints [http://minio-etcd.minio:2379/] (*fmt.wrapError)
       1: cmd/iam.go:629:cmd.(*IAMSys).Init()

API: SYSTEM()
Time: 20:22:51 UTC 08/04/2021
DeploymentID: d28fda90-0399-453a-9fe3-fc83921ef50c
Error: IAM sub-system is partially initialized, some users may not be available (*errors.errorString)
       1: cmd/iam.go:630:cmd.(*IAMSys).Init()

If I make a curl to the etcd pod inside the minIO pod, I can connect with that properly.

Command: curl http://minio-etcd.minio:2379/version

{"etcdserver":"3.5.0","etcdcluster":"3.5.0"}

The endpoint http://minio-etcd.minio:2379/ isn't available, I don't know if it is the error.

404 page not found

Steps to Reproduce (for bugs)

I deployed both minIO and etcd using the bitnami helm charts.

apiVersion: v1
kind: Namespace
metadata:
  name: minio
  labels:
    istio.io/rev: 1-10-2
---
apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: etcd
  namespace: minio
spec:
  helmVersion: v3
  chart:
    repository: https://charts.bitnami.com/bitnami
    name: etcd
    version: 6.2.11
  values:
    commonLabels:
      app: minio-etcd
      version: 3.5.0
    persistence:
      enabled: true
      storageClass: efs
      accessMode: ReadWriteMany
      size: 10Gi
    diagnosticMode:
      enabled: true
    podLabels:
      app: minio-etcd
      version: 3.5.0
---
apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
    name: minio
    namespace: minio
spec:
    helmVersion: v3
    chart:
        repository: https://charts.bitnami.com/bitnami
        name: minio
        version: 7.1.6
    values:
        global:
          minio:
            existingSecret: minio-secret
        existingSecret: minio-secret
        mode: distributed
        persistence:
          enabled: false
        serviceAccount:
          name: minio-svc-account
          annotations:
            eks.amazonaws.com/role-arn: <AN-AMAZING-AWS-ARN-ROLE-HERE>
        gateway:
          sse:
            enabled: false
          enabled: true
          type: s3
          auth:
            s3:
              useIAMRole: true
        containerPort: 9000
        extraEnv:
          - name: MINIO_IDENTITY_OPENID_CONFIG_URL
            value: http://<KEYCLOAK-NAME>.<KEYCLOAK-NAMESPACE>/auth/realms/<KEYCLOAK-REALM>/.well-known/openid-configuration
          - name: MINIO_IDENTITY_OPENID_REDIRECT_URI
            value: https://<MINIO-ENDPOINT>/oauth_callback
          - name: MINIO_IDENTITY_OPENID_CLAIM_NAME
            value: minio_policies
          - name: MINIO_IDENTITY_OPENID_CLIENT_ID
            value: minio
          - name: MINIO_IDENTITY_OPENID_CLIENT_SECRET
            valueFrom:
              secretKeyRef:
                name: minio-auth-secret
                key: client_secret
          - name: MINIO_IDENTITY_OPENID_VENDOR
            value: keycloak
          - name: MINIO_IDENTITY_OPENID_KEYCLOAK_REALM
            value: master
          - name: MINIO_IDENTITY_OPENID_KEYCLOAK_ADMIN_URL
            value: https://<KEYCLOAK-EXTERNAL-URL>
          - name: MINIO_IDENTITY_OPENID_SCOPES
            value: openid,email,minio_policies,profile
          - name: MINIO_ETCD_ENDPOINTS
            value: http://minio-etcd.minio:2379/

Your Environment

System	Version
Istio	1.10.2
FluxCD	v1
bitnami/etcd	3.5.0-debian-10-r42
bitnami/minio	2021.6.17-debian-10-r37
bitnami/minio-client	2021.7.27-debian-10-r7
Keycloak	14.0.0

The text was updated successfully, but these errors were encountered:

harshavardhana · 2021-08-04T21:05:19Z

You purge your entire etcd and re-deploy it should work fine. @DougTrajano

DougTrajano · 2021-08-05T00:05:17Z

You purge your entire etcd and re-deploy it should work fine. @DougTrajano

I deleted all the resources related to my minio-etcd and re-deploy it again, but it stays in the same issue. :/

harshavardhana · 2021-08-05T00:15:23Z

I deleted all the resources related to my minio-etcd and re-deploy it again, but it stays in the same issue. :/

AFAICS @DougTrajano etcd is not taking any writes we are trying to create a key on it and that's failing - you need to first check if etcd is taking writes or at least how much time it takes to write a key into etcd.

It looks like it's beyond 30secs - since that is the deadline we have for our individual etcd API calls.

DougTrajano · 2021-08-05T17:09:12Z

I deleted all the resources related to my minio-etcd and re-deploy it again, but it stays in the same issue. :/

AFAICS @DougTrajano etcd is not taking any writes we are trying to create a key on it and that's failing - you need to first check if etcd is taking writes or at least how much time it takes to write a key into etcd.

It looks like it's beyond 30secs - since that is the deadline we have for our individual etcd API calls.

I'll check that

DougTrajano · 2021-08-06T18:34:30Z

Hey buddy, sorry for the late answer, I did some tests and changes in my deployment manifest.

helm charts

apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
    name: minio
    namespace: minio
spec:
    helmVersion: v3
    chart:
        repository: https://charts.bitnami.com/bitnami
        name: minio
        version: 7.1.6
    values:
        global:
          minio:
            existingSecret: minio-secret
        existingSecret: minio-secret
        mode: distributed
        persistence:
          enabled: false
        serviceAccount:
          name: minio-svc-account
          annotations:
            eks.amazonaws.com/role-arn: <AWS-ARN>
        gateway:
          sse:
            enabled: false
          enabled: true
          type: s3
          auth:
            s3:
              useIAMRole: true
        containerPort: 9000
        extraEnv:
          - name: MINIO_IDENTITY_OPENID_CONFIG_URL
            value: <KEYCLOAK-INTERNAL-URL>/realms/<KEYCLOAK-REALM>/.well-known/openid-configuration
          - name: MINIO_IDENTITY_OPENID_REDIRECT_URI
            value: <KEYCLOAK-INTERNAL-URL>/minio/oauth_callback
          - name: MINIO_IDENTITY_OPENID_CLAIM_NAME
            value: minio_policies
          - name: MINIO_IDENTITY_OPENID_CLIENT_ID
            value: minio
          - name: MINIO_IDENTITY_OPENID_CLIENT_SECRET
            valueFrom:
              secretKeyRef:
                name: minio-auth-secret
                key: client_secret
          - name: MINIO_IDENTITY_OPENID_VENDOR
            value: keycloak
          - name: MINIO_IDENTITY_OPENID_KEYCLOAK_REALM
            value: <KEYCLOAK-REALM>
          - name: MINIO_IDENTITY_OPENID_KEYCLOAK_ADMIN_URL
            value: <KEYCLOAK-EXTERNAL-URL>
          - name: MINIO_IDENTITY_OPENID_SCOPES
            value: openid,email,minio_policies,profile
          - name: MINIO_ETCD_ENDPOINTS
            value: http://minio-etcd.minio.svc.cluster.local:2379
          - name: MINIO_HTTP_TRACE
            value: /dev/stdout
---
apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: etcd
  namespace: minio
spec:
  helmVersion: v3
  chart:
    repository: https://charts.bitnami.com/bitnami
    name: etcd
    version: 6.3.3
  values:
    image:
      registry: <MY-REGISTRY>
      repository: bitnami/etcd
      tag: 3.5.0-debian-10-r42
    auth:
      rbac:
        enabled: false
        rootPassword: root
        allowNoneAuthentication: true
    commonLabels:
      app: minio-etcd
      version: 3.5.0
    persistence:
      enabled: true
      storageClass: efs
      accessMode: ReadWriteMany
      size: 20Gi
    podLabels:
      app: minio-etcd
      version: 3.5.0

Then, I did several troubleshooting steps that I'll describe below.

Troubleshooting steps

etcd configs

...
auth:
  rbac:
    enabled: false
    rootPassword: root
    allowNoneAuthentication: true
...

etcd pod: export ETCD_ROOT_USERNAME=root
etcd pod: etcdctl put foo "bar"
- Return: OK - SUCCESS
minio pod: Install etcdctl
minio pod: $ETCD_BIN/etcdctl put greeting "Hello, etcd" --endpoints=http://minio-etcd.minio:2379
- Return: FAILED

{
    "level": "warn",
    "ts": "2021-08-06T17:55:34.117Z",
    "logger": "etcd-client",
    "caller": "v3/retry_interceptor.go:62",
    "msg": "retrying of unary invoker failed",
    "target": "etcd-endpoints://0xc00010e380/#initially=[http://minio-etcd.minio:2379]",
    "attempt": 0,
    "error": "rpc error: code = Unavailable desc = error reading from server: EOF"
}

minio pod: $ETCD_BIN/etcdctl put greeting "Hello, etcd" --endpoints=http://minio-etcd.minio:2379 --user="root:root"
- Return: FAILED

{
    "level": "warn",
    "ts": "2021-08-06T17:46:10.232Z",
    "logger": "etcd-client",
    "caller": "v3/retry_interceptor.go:62",
    "msg": "retrying of unary invoker failed",
    "target": "etcd-endpoints://0xc000620000/#initially=[http://minio-etcd.minio.svc.cluster.local:2379]",
    "attempt": 0,
    "error": "rpc error: code = Unavailable desc = error reading from server: read tcp XXX.XX.XXX.XX:XXXXX->XX.XXX.XXX.XX:2379: read: connection reset by peer"
}

Error: rpc error: code = Unavailable desc = error reading from server: read tcp XXX.XX.XXX.XX:XXXXX->XX.XXX.XXX.XX:2379: read: connection reset by peer

minio pod: $ETCD_BIN/etcdctl endpoint health
- Return: FAILED

{
    "level": "warn",
    "ts": 999999999,
    "logger": "client",
    "caller": "v3/retry_interceptor.go:62",
    "msg": "retrying of unary invoker failed",
    "target": "etcd-endpoints://0xc00031e700/#initially=[127.0.0.1:2379]",
    "attempt": 0,
    "error": "rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\""
}

etcd pod: etcdctl endpoint health
- Return: 127.0.0.1:2379 is healthy: successfully committed proposal: took = 1.525918ms
minio pod: curl -L http://minio-etcd.minio:2379/v3/kv/put -X POST -d '{"key": "Zm9v", "value": "YmFy"}'
- Return: SUCCESS (I can write in etcd using curl)

{
    "header": {
        "cluster_id": "XX",
        "member_id": "XX",
        "revision": "4",
        "raft_term": "2"
    }
}

minio pod curl -L http://minio-etcd.minio:2379/version
- Return: SUCCESS

{
    "etcdserver": "3.5.0",
    "etcdcluster": "3.5.0"
}

DougTrajano · 2021-08-06T18:37:47Z

It seems that I can write to etcd using curl, but etcdctl doesn't work. :(

DougTrajano · 2021-08-06T19:23:31Z

Downgrade etcd version (from 3.5.0-debian-10-r42 to 3.3.9-debian-9-r73)

minio pod: curl -L http://minio-etcd-0.minio-etcd-headless.minio.svc.cluster.local:2379/v3/kv/put -X POST -d '{"key": "Zm9v", "value": "YmFy"}'
- Return: 404 page not found
minio pod: curl -L http://minio-etcd-0.minio-etcd-headless.minio.svc.cluster.local:2379/kv/put -X POST -d '{"key": "Zm9v", "value": "YmFy"}'
- Return: 404 page not found
minio pod: curl -L http://minio-etcd-0.minio-etcd-headless.minio.svc.cluster.local:2379/v3alpha/kv/put -X POST -d '{"key": "Zm9v", "value": "YmFy"}'X
- Return: SUCCESS

{
    "header": {
        "cluster_id": "XXX",
        "member_id": "XXX",
        "revision": "3",
        "raft_term": "6"
    }
}

minio pod: curl -L http://minio-etcd-0.minio-etcd-headless.minio.svc.cluster.local:2379/v3alpha/kv/put -X POST -d '{"key": "Zm9v", "value": "YmFy"}'X
- Return: SUCCESS

{
    "header": {
        "cluster_id": "XXX",
        "member_id": "XXX",
        "revision": "3",
        "raft_term": "9"
    },
    "kvs": [
        {
            "key": "Zm9v",
            "create_revision": "2",
            "mod_revision": "3",
            "version": "2",
            "value": "YmFy"
        }
    ],
    "count": "1"
}

minio pod: curl -L http://minio-etcd-0.minio-etcd-headless.minio.svc.cluster.local:2379/v3beta/kv/range -X POST -d '{"key": "Zm9v"}'
- Return: SUCCESS

{
    "header": {
        "cluster_id": "XXX",
        "member_id": "XXX",
        "revision": "3",
        "raft_term": "8"
    },
    "kvs": [
        {
            "key": "Zm9v",
            "create_revision": "2",
            "mod_revision": "3",
            "version": "2",
            "value": "YmFy"
        }
    ],
    "count": "1"
}

Conclusions

etcd has critical changes between the patch versions, someone knows what is a stable version tested within minio?

harshavardhana · 2021-08-06T23:41:01Z

etcd has critical changes between the patch versions, someone knows what is a stable version tested within minio?

etcd v3.3 and v3.4 - I have tested, not sure v3.5

DougTrajano · 2021-08-09T14:48:31Z

Hey, @harshavardhana I did some more tests.

Actually, the most important test that I did is create a Docker Compose to validate the Bitnami images and application versions. All of these works fine on that. For someone that wants to check the Docker Compose, I added this to a repository on my GitHub.

github.com/DougTrajano/docker-compose-examples/minio-sts

So, I'm still facing this issue yet, but I think that this is not related to any of the applications, is something in my environment. If you want to close this issue, I'll post here the solution (when I find it) to help in the future.

Ps: I used etcd v.3.5 and works fine! :)

harshavardhana · 2021-08-09T16:25:41Z

Hey, @harshavardhana I did some more tests.

Actually, the most important test that I did is create a Docker Compose to validate the Bitnami images and application versions. All of these works fine on that. For someone that wants to check the Docker Compose, I added this to a repository on my GitHub.

github.com/DougTrajano/docker-compose-examples/minio-sts

So, I'm still facing this issue yet, but I think that this is not related to any of the applications, is something in my environment. If you want to close this issue, I'll post here the solution (when I find it) to help in the future.

Ps: I used etcd v.3.5 and works fine! :)

👍🏽

DougTrajano · 2021-08-16T18:50:06Z

@harshavardhana we founded the reason. It's the same issue that was described in #13192.

When I downgrade minIO to an old version (2021.5.27-debian-10-r8) it works perfectly.

As I have Istio (Envoy Proxy) in my Kubernetes Cluster the requests are refusing because the :authority is invalid.

DougTrajano added community triage labels Aug 4, 2021

harshavardhana closed this as completed Aug 9, 2021

harshavardhana added working as intended and removed triage labels Aug 9, 2021

github-actions bot locked as resolved and limited conversation to collaborators Aug 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

etcd is unreachable, please check your endpoints #12877

etcd is unreachable, please check your endpoints #12877

DougTrajano commented Aug 4, 2021 •

edited

harshavardhana commented Aug 4, 2021

DougTrajano commented Aug 5, 2021

harshavardhana commented Aug 5, 2021 •

edited

DougTrajano commented Aug 5, 2021

DougTrajano commented Aug 6, 2021 •

edited

DougTrajano commented Aug 6, 2021

DougTrajano commented Aug 6, 2021

harshavardhana commented Aug 6, 2021

DougTrajano commented Aug 9, 2021 •

edited

harshavardhana commented Aug 9, 2021

DougTrajano commented Aug 16, 2021 •

edited

etcd is unreachable, please check your endpoints #12877

etcd is unreachable, please check your endpoints #12877

Comments

DougTrajano commented Aug 4, 2021 • edited

Expected Behavior

Current Behavior

Steps to Reproduce (for bugs)

Your Environment

harshavardhana commented Aug 4, 2021

DougTrajano commented Aug 5, 2021

harshavardhana commented Aug 5, 2021 • edited

DougTrajano commented Aug 5, 2021

DougTrajano commented Aug 6, 2021 • edited

Troubleshooting steps

etcd configs

DougTrajano commented Aug 6, 2021

DougTrajano commented Aug 6, 2021

Downgrade etcd version (from 3.5.0-debian-10-r42 to 3.3.9-debian-9-r73)

Conclusions

harshavardhana commented Aug 6, 2021

DougTrajano commented Aug 9, 2021 • edited

harshavardhana commented Aug 9, 2021

DougTrajano commented Aug 16, 2021 • edited

DougTrajano commented Aug 4, 2021 •

edited

harshavardhana commented Aug 5, 2021 •

edited

DougTrajano commented Aug 6, 2021 •

edited

DougTrajano commented Aug 9, 2021 •

edited

DougTrajano commented Aug 16, 2021 •

edited