[etcd] Bump etcd client to 3.5.1

[ahrtr]

What would you like to be added?

When upgrading etcd from an old version to 3.5.0, then some zombie members may be displayed. Users can't even remove the zombie members using command etcdctl member remove <id>. Please see the discussion in etcd/issues/13196.

A fix for this issue has already been included in etcd 3.5.1. So it'd be better to bump etcd 3.5.1, and cherry pick to 1.22.

Why is this needed?

Once etcd is upgraded to 3.5.1, then the zombie members can be removed either automatically or manually.

I see that PR pull/105706 fixed this, but the all the go.mod files are still referencing to etcd 3.5.0.

[k8s-ci-robot]

@ahrtr: The label(s) sig/etcd cannot be applied, because the repository doesn't have them.

In response to this:

/sig etcd

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Kartik494]

Hi @ahrtr i am looking into this. Please let me know if 105706 also needs to be backported in v1.22

[Kartik494]

/assign

[neolit123]

we are in code freeze for 1.23 so this must happen for 1.24.

/milestone v1.24
/sig cluster-lifecycle release
/area etcd

[neolit123]

we are in code freeze for 1.23 so this must happen for 1.24.

unless this is considered a critical bug without workarounds?

EDIT

[neolit123]

I see that PR pull/105706 fixed this, but the all the go.mod files are still referencing to etcd 3.5.0.

right, we updated the server to 3.5.1 recently (i forgot).
but the go.mod files are indeed out of date, which would mean we are still using the 3.5.0 client.
matching server/client seems reasonable for 1.23.

tagging for release triage.

/triage accepted
/milestone v1.23
/priority important-soon

[neolit123]

/retitle [etcd] Bump etcd client to 3.5.1

[neolit123]

@ahrtr if we have a 3.5.0 client and a 3.5.1 server is this still a problem?

[ahrtr]

@neolit123 There is an important fix (see below) being included in etcd 3.5.1 on v3 client,
client: Use first endpoint as http2 authority header
Cherry pick "Fix http2 authority header in single endpoint scenario" to release-3.5

The related issue is etcd/issues/13192.

I think @serathius is the best person to answer this question.

cc @uthark

[serathius]

v3.5.1 client includes fix for authority header in HA cluster. Without the fix, client will send invalid authority header when configured with multiple endpoints. This is not a problem when client communicates directly to etcd server, however will not work at all if there is any proxy before etcd. If there is a proxy before etcd there is a high chance that, invalid authority header will result in requests being dropped. This was deemed a critical bug for v3.5.0 as it totally broke some multi node etcd configurations.

[neolit123]

Thanks for the explanation.
/priority critical-urgent

I guess this means we need to backport the client bump to 1.22.

[ahrtr]

Thanks for the explanation. /priority critical-urgent

I guess this means we need to backport the client bump to 1.22.

The PR pull/105706 is not included in 1.22, so we need to backport it to 1.22, and also bump etcd client to 3.5.1 for both 1.23 and 1.22.

[Kartik494]

@neolit123 should we wait for #106591 to land in master or need to backport #105706 in v1.22 now on immediate basis ?

[neolit123]

This seems like a change that should be part of 1.23 before release and backported to older releases. Although our HA upgrade e2e tests are not exhibiting the bug for some reason.

[Kartik494]

@neolit123 Could you please confirm whether the #106591 is planned after the v1.23 release ?

[ritpanjw]

Hi @Kartik494 , this is bug triage shadow here 👋
I'd like to check what's the status of this issue, release 1.23 would be happening this week
Thank you

[neolit123]

Looks like the release team did not want this last minute change in 1.23.0. Backporting it might need discussion.. . Seems fine for 1.24. On Nov 29, 2021 10:24, "Kartik Sharma" ***@***.***> wrote: @neolit123 <https://github.com/neolit123> Could you please confirm whether the #106591 <#106591> is planned after the v1.23 release ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#106589 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACRATDTK6B6UMBWZKZKQSDUOM2EVANCNFSM5IQFICOA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[peterska]

Can this backported to k8s 1.22.X please. My cluster will not upgrade from 1.21.7 to 1.22.4 due to this issue. It gets stuck waiting for etcd to be ready. Checking kube-system/etcd logs shows it is trying to contact the zombie etcd node. This is a single node etcd kubernetes cluster created using kubeadm a long time ago. I migrated the cluster so that the api sever is accessed using a dns name rather than an ip address and have since changed the ip address . This what caused the phantom etcd member.

[pacoxu]

• Opened cherry-pick pr Automated cherry pick of #105706: Upgrade etcd to 3.5.1 #108176 for 1.23. (It only backports the server fix for etcd 3.5.0 resurrects ancient (unremovable) members etcd-io/etcd#13196).
• To backport client change in 3.5.1 as well. We need to cherry-pick Updated Etcd Version to 3.5.1 in go.mod #106591 to 1.23 & 1.22 after it is merged.

[Kartik494]

@pacoxu Could you please let me know if #106591 target for 1.24 release?

[pacoxu]

@pacoxu Could you please let me know if #106591 target for 1.24 release?

I think it should. But it need confirmation from Jordon and Marek. See discussions in #106591 (comment).

[Kartik494]

Thanks for the clarification !

[neolit123]

Should we wait for this 3.5 backport as well? etcd-io/etcd#13706 Xref kubernetes/kubeadm#2567

[akunszt]

If this won't be to 1.22 not 1.23 then what is the official upgrade path? We are running 1.21 in an environment where we access the etcd cluster through a proxy. Due to the broken etcd client in those version we can't use neither 1.22 nor 1.23, they can't connect to the etcd at all. As far as I know upgrading directly to 1.24 is not supported. How can we escape from this trap?

[neolit123]

we have kubeadm HA cluster upgrade tests from 1.21 -> 1.22 -> 1.23 -> latest and these are all green.
https://k8s-testgrid.appspot.com/sig-cluster-lifecycle-kubeadm

so oddly we are not catching any of these reported problems. it's also not clear if the reporting users are:

• reporting separate problems in etcd (client/server?)
• using kubeadm or something else

kubeadm embeds an etcd client, so this means we have to backport a fix for kubeadm.
if you are not using kubeadm and e.g. directly using etcdctl or something else you'd have to update that separate tool with a patched client that works.

[akunszt]

We don't use kubeadm. We have an ALB in front of the etcd cluster as using a DNS based discovery wasn't - or isn't - working in AWS environment properly.

The apiserver container fails to connect to the etcd cluster. We would like to update that "tool".

[neolit123]

ok, forgot the apiserver has the same client too (duh).
instead of 3.5.1 i think we should get the latest 3.5.x (.3 once it's out?) and backport it to the supported k8s versions.
is this all about etcd-io/etcd#13196 or are there other etcd client bugs in question here?

cc @kubernetes/sig-api-machinery-bugs

[akunszt]

@neolit We suffer from the etcd-io/etcd#13192 issue. I think those two had the same root cause though. (I'm not an etcd developer, so that's just my hunch.)

[neolit123]

@pacoxu @serathius current state of the pending changes is a bit messy here.
i personally think we should drop the 3.5.1 PRs and backport PRs and should wait for 3.5.3.
...but given there are a number of users here than want to upgrade ASAP, we can upgrade client/server to 3.5.2 first.
EDIT: looks like #105706 merged and the client / server at master are supposedly at .1. so we now need > 3.5.1. the client is still at 3.5.0?

i don't like upgrading only client separately from server. i think we should keep them in sync.
so we can rename / repurpose this issue.

[neolit123]

looks like the 3.5.1 client bump is blocked here:
#106591 (review)
due to etcd-io/etcd#13707

but from discussion on this 3.5.2 server PR, people already want a 3.5.2:
#107917

[akunszt]

@neolit123 Honestly, anything above 3.5.0 would make me smile. So 3.5.2 is even better.

[ahrtr]

looks like the 3.5.1 client bump is blocked here: #106591 (review) due to etcd-io/etcd#13707

but from discussion on this 3.5.2 server PR, people already want a 3.5.2: #107917

I have already submitted a PR etcd/pull/13737 for etcd/issues/13707. cc @serathius @ptabor

[Kartik494]

Hi @neolit123 as #106591 has been merged, so can we close this issue?
Thanks!!

[neolit123]

Yes, we can close this but we would need a tracking issue for the inbound 3.5.3 bump and we may have to backport it (at least the server bump). Ideally we should have separate tracking for client / server. /close

…

On Mar 31, 2022 09:37, "Kartik Sharma" ***@***.***> wrote: Hi @neolit123 <https://github.com/neolit123> as #106591 <#106591> has been merged, so can we close this issue? — Reply to this email directly, view it on GitHub <#106589 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACRATAL6VSPUF3RTUHW3IDVCVB2HANCNFSM5IQFICOA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[k8s-ci-robot]

@neolit123: Closing this issue.

In response to this:

Yes, we can close this but we would need a tracking issue for the inbound
3.5.3 bump and we may have to backport it (at least the server bump).

Ideally we should have separate tracking for client / server.

/close
On Mar 31, 2022 09:37, "Kartik Sharma" @.***> wrote:

Hi @neolit123 https://github.com/neolit123 as #106591
#106591 has been merged,
so can we close this issue?

—
Reply to this email directly, view it on GitHub
#106589 (comment),
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACRATAL6VSPUF3RTUHW3IDVCVB2HANCNFSM5IQFICOA
.
You are receiving this because you were mentioned.Message ID:
@.***>

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kopiczko]

Is upgrade to newer etcd version tracked somewhere? According to this 3.5.1 is not production-grade due to data corruption.

[pacoxu]

In v1.25, #110033 already uses 3.5.4. And cherry-pick to v1.24-v1.22 are opened for reviewing.