OCPBUGS-947: Rebase openshift/etcd 4.11 onto 3.5.5 #154

…s, as they are taken care by hook. missing after 3.5.0 merge

missing after merge of v3.5.0 tag Thanks to this change: - all the maps bucket -> buffer are indexed by int's instead of string. No need to do: byte[] -> string -> hash conversion on each access. - buckets are strongly typed in backend/mvcc API.

…to a role Missing after v3.5.0 tag merge Prevent etcd from crashing when given a bad grant payload, e.g.: $ curl -d '{"name": "foo"}' http://localhost:2379/v3/auth/role/add {"header":{"cluster_id":"14841639068965178418", ... $ curl -d '{"name": "foo"}' http://localhost:2379/v3/auth/role/grant curl: (52) Empty reply from server Signed-off-by: Gyuho Lee <leegyuho@amazon.com>

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>

… is a distributed" Correcting a typo from 2f10964 (version: openshift-v4.0, 2018-11-29).

…operly copied from build. Signed-off-by: Sam Batschelet <sbatsche@redhat.com>

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>

…se-3.3 Signed-off-by: Sam Batschelet <sbatsche@redhat.com>

make build performs a sanity test on the binary image which causes problems for unsupport arch. Because we run full CI tests against the image this check is not nessisary and will allow images to be build regardless of arch. Signed-off-by: Sam Batschelet <sbatsche@redhat.com>

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>

…ember but memberDir exists, go ahead and start.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>

…dling when we dont scale member Signed-off-by: Sam Batschelet <sbatsche@redhat.com>

…nsistent with ART Reconciling with https://github.com/openshift/ocp-build-data/tree/990044f295fb1d5e238823902962dbcfa1c041c9/images/ose-etcd.yml

…nsistent with ART Reconciling with https://github.com/openshift/ocp-build-data/tree/ac81dd4ff0bd57c4e75058d25b40615b92948259/images/ose-etcd.yml

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>

…nsistent with ART Reconciling with https://github.com/openshift/ocp-build-data/tree/5a1293dd0f380abf50c12d65c36655486d7745d0/images/ose-etcd.yml

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>

…s not part of member list and dataDir exists Signed-off-by: Sam Batschelet <sbatsche@redhat.com>

Reconciling with https://github.com/openshift/ocp-build-data/tree/691e628254f318ce56efda5edc7448ec743c37b8/images/ose-etcd.yml

Reconciling with https://github.com/openshift/ocp-build-data/tree/e8b7c67cc73ae9b0dbf97f61bde7953f8f79bc21/images/ose-etcd.yml

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>

…r when its cert has non empty CN CARRY: This patch is required to allow metrics cert to auth with CN in cert. Signed-off-by: Sam Batschelet <sbatsche@redhat.com>

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>

…nfig file

…on tests with unit etcd 3.5 will not allow integration tests to run outside of test/integration. This change breaks out the logic into a more testable path allowing coverage and not requiring breaking out discover-etcd-initial-cluster into a seperate module yet. Signed-off-by: Sam Batschelet <sbatsche@redhat.com>

ETCD-202: bump etcd v3.5.0

Reconciling with https://github.com/openshift/ocp-build-data/tree/399029b7ca4afdc9f966d18d09e27a3ac0400bfb/images/ose-etcd.yml

…of-#13145-etcd-io#13237-origin-release-3.5 [backport 3.5]: Automated cherry pick of etcd-io#13145 etcd-io#13237

…shift-4.9-ose-etcd ETCD-203: Updating ose-etcd images to be consistent with ART

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>

….art and update for cachito Signed-off-by: Sam Batschelet <sbatsche@redhat.com>

Bug 1992673: Fix build

This is often used by both support and end users, which causes etcd to get load sometimes in live production env.

Bug 1994986: (CARRY) etcdctl/ctlv3/ctl.go: Remove check perf command

Stop using tip golang version in CI

…ckend out of sync

Reconciling with https://github.com/openshift/ocp-build-data/tree/3dea8426481b269476342fc0a631273f32ce691e/images/ose-etcd.yml

Fix for v3.5 Ensure that cluster members stored in v2store and backend are in sync

…shift-4.10-ose-etcd Updating ose-etcd images to be consistent with ART

prefixArgs uses os.Setenv in e2e tests instead envMap. This creates overwrites in some test cases and have an impact on test quality and isolation between tests. This PR uses ctlcontext envMap in each tests with high priority and merges os environment variables with low priority.

Cherry pick "Fix http2 authority header in single endpoint scenario" to release-3.5

fixes: CVE-2021-3711, CVE-2021-35942, CVE-2019-9893 Signed-off-by: Sam Batschelet <sbatsche@redhat.com>

[release-3.5] Dockerfile: bump debian bullseye-20210927

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>

…t from 3.6)

storage/backend: Add a gauge to indicate if defrag is active (backport)

… is active

UPSTREAM: <carry>: storage/backend: Add a gauge to indicate if defrag is active

…tored in v2store and backend are in sync (cherry picked from commit e68c7ab)

…-store-member-sync Bug 2003775: UPSTREAM: <carry>: server: Fix for v3.5 Ensure that cluster members stored in v2store and backend are in sync

… by multiple peers

cherry-pick to 3.5 from etcd-io#13467 exclude the same alarm type activated by multiple peers

…ying To fix a bug in the retry logic caused when the auth token is cleared after receiving `ErrInvalidAuthToken` from the server and the subsequent call to `getToken` also fails due to some reason (eg. context deadline exceeded). This leaves the client without a token and the retry will continue to fail with `ErrUserEmpty` unless the token is refreshed.

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>

OWNERS: clean up

UPSTREAM: <carry>: add --experimental-max-learner flag

Backport PR 13308 to release-3.5

Current checkpointing mechanism is buggy. New checkpoints for any lease are scheduled only until the first leader change. Added fix for that and a test that will check it.

To extend lease checkpointing mechanism to cases when the whole etcd cluster is restarted.

…lease-checkpoint-persist to persist lease remainingTTL To avoid inconsistant behavior during cluster upgrade we are feature gating persistance behind cluster version. This should ensure that all cluster members are upgraded to v3.6 before changing behavior. To allow backporting this fix to v3.5 we are also introducing flag --experimental-enable-lease-checkpoint-persist that will allow for smooth upgrade in v3.5 clusters with this feature enabled.

Backport Lease Checkpoints fix to release-3.5

[3.5] Set the backend again after recovering v3 backend from snapshot

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>

OWNERS: bump

…ock-fix Backport watchablestore runlock bug fix to release-3.5

…9-11254.

Update dep: gopkg.in/yaml.v2 v2.2.8 -> v2.4.0 due to: CVE-2019-11254 [release 3.5]

To rectify the vulnerability found in a version of golang.org/x/crypto (https://avd.aquasec.com/nvd/cve-2020-29652), upgrade the dependency to its latest version. Alternatively, version v0.0.0-20201216223049-8b5274cf687f could be used, where the fixed was introduced, but the latest is preferable.

Problem: Defrag was implemented before custom bolt options were added. Currently defrag doesn't restore backend options. For example BackendFreelistType will be unset after defrag. Solution: save bolt db options and use them in defrag.

[3.5] Trim the suffix dot from the srv.Target for etcd-client DNS lookup

…heck [3.5] enhance health check endpoint to support serializable request

porting back from 3.5 from main PR etcd-io#13334

*: fix IsOptsWithFromKey

…cimal [3.5] Always print raft term in decimal when displaying member list in json

server/storage/backend: restore original bolt db options after defrag

backport 3.5: etcd-io#13676 load all leases from backend

Fix offline defrag in etcdctl

This should make it easier to bump the golang version

[release-3.5] Update go to 1.16.15

v3.5.2

Signed-off-by: Manuel Rüger <manuel@rueg.eu>

[Release-3.5] Makefile: Fix wrong target

This fixes problem where logs json changes format of timestamp.

Contiguous to 3.5.2

[release-3.5] Fix inconsistent log format

…e of apply

[release-v3.5] server: Add verification of whether lock was called within out outsid…

[release-3.5] go.mod: Upgrade to prometheus/client_golang v1.11.1

…rease Reason to store CI and term in backend was to make db fully independent snapshot, it was never meant to interfere with apply logic. Skip of CI was introduced for v2->v3 migration where we wanted to prevent it from decreasing when replaying wal in etcd-io#5391. By mistake it was added to apply flow during refactor in etcd-io#12855 (comment). Consistency index and term should only be negotiated and used by raft to make decisions. Their values should only driven by raft state machine and backend should only be responsible for storing them.

[release-3.5] server: Save consistency index and term to backend even when they decrease

…backend Previously the SetConsistentIndex() is called during the apply workflow, but it's outside the db transaction. If a commit happens between SetConsistentIndex and the following apply workflow, and etcd crashes for whatever reason right after the commit, then etcd commits an incomplete transaction to db. Eventually etcd runs into the data inconsistency issue. In this commit, we move the SetConsistentIndex into a txPostLockHook, so it will be executed inside the transaction lock.

[3.5] Fix the data inconsistency issue by adding a txPostLockHook into the backend

…pendency-golang.org/x/crypto etcdserver: upgrade the golang.org/x/crypto dependency

This change is to ensure that all members returned during the client's AutoSync are started and are not learners, which are not valid etcd members to make requests to.

[release-3.5] clientv3: filter learners members during autosync

When etcdserver receives a LeaseRenew request, it may be still in progress of processing the LeaseGrantRequest on exact the same leaseID. Accordingly it may return a TTL=0 to client due to the leaseID not found error. So the leader should wait for the appliedID to be available before processing client requests.

[3.5] Support linearizable renew lease

Signed-off-by: Manuel Rüger <manuel@rueg.eu>

…ex_3.5 [3.5]Set backend to cindex before recovering the lessor in applySnapshot

[release-3.5] Dockerfile*: Switch baseimage to k8s hosted one

When a user sets up a Mirror with a restricted user that doesn't have access to the `foo` path, we will fail to get the most recent revision due to permissions issues. With this change, when a prefix is provided we will get the initial revision from the prefix rather than /foo. This allows restricted users to setup sync.

[backport] PR 13923 to release-3.5

v3.5.3

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>

3.5.3 with history

…ookup" This reverts commit 4f51cc1.

Revert etcd-io#13714

When clients have no permission to perform whatever operation, then the applying may fail. We should also move consistent_index forward in this case, otherwise the consitent_index may smaller than the snapshot index.

…_353 [3.5] Update consitent_index when applying fails

This reverts commit 6eceab4.

…embers to start without quorum for recovery scenarios

Remove MemberList patch

Update crypto to address CVE-2022-27191. The CVE fix is added in 0.0.0-20220315160706-3147a52a75dd but this change updates to latest.

Update golang.org/x/crypto to latest

we found a lease leak issue: if a new member(by member add) is recovered by snapshot, and then become leader, the lease will never expire afterwards. leader will log the revoke failure caused by "invalid auth token", since the token provider is not functional, and drops all generated token from upper layer, which in this case, is the lease revoking routine.

[backport 3.5]: server/auth: enable tokenProvider if recoved store enables auth

Co-authored-by: Haseeb Tariq <hasbro17@gmail.com>

Add new members to reviewers/approvers

[release-3.5] Test release scripts

This PR removes additional clone when building artifacts. When releasing v3.5.4 this clone was main cause of issues and confusion about what release script is doing. release.sh script already clones repo in /tmp/ directory, so clonning before build is not needed. As precautions for bug in script leaving /tmp/ clone in bad state I moved "Verify the latest commit has the version tag" and added "Verify the clean working tree" to be always run before build.

[release-3.5] scripts: Avoid additional repo clone

it gives more time to the etcd operator to add a new member. the etcd operator expects the etcd container to be in a running state. too tight timeout will cause the container to restart and decreasing operator chances

…e-disco-2 Bug 2085997: increases cluster discovery time from 10s to 180s

…to 180s" This reverts commit a66d3c3.

Revert "UPSTREAM: <carry>: increases cluster discovery time from 10s to 180s"

it gives more time to the etcd operator to add a new member. the etcd operator expects the etcd container to be in a running state. too tight timeout will cause the container to restart and decreasing operator chances

…e-disco-3 Bug 2085997: increases cluster discovery time from 10s to 135s

The first bug fix is to resolve the race condition between goroutine and channel on the same leases to be revoked. It's a classic mistake in using Golang channel + goroutine. Please refer to https://go.dev/doc/effective_go#channels The second bug fix is to resolve the issue that etcd lessor may continue to schedule checkpoint after stepping down the leader role.

[3.5] Backport two lease related bug fixes to 3.5

The FileReader interface is the wrapper of io.Reader. It provides the fs.FileInfo as well. The FileBufReader struct is the wrapper of bufio.Reader, it also provides fs.FileInfo. Signed-off-by: Benjamin Wang <wachao@vmware.com>

…file Currently the max size of each WAL entry is hard coded as 10MB. If users set a value > 10MB for the flag --max-request-bytes, then etcd may run into a situation that it successfully processes a big request, but fails to decode it when replaying the WAL file on startup. On the other hand, we can't just remove the limitation, because if a WAL entry is somehow corrupted, and its recByte is a huge value, then etcd may run out of memory. So the solution is to restrict the max size of each WAL entry as a dynamic value, which is the remaining size of the WAL file. Signed-off-by: Benjamin Wang <wachao@vmware.com>

[3.5] Restrict the max size of each WAL entry to the remaining size of the WAL file

Cherry pick the PR etcd-io#12992 to 3.5, so please refer to the original PR for more detailed info. Signed-off-by: Benjamin Wang <wachao@vmware.com>

[3.5] client/v3: do not overwrite authTokenBundle on dial

Make sure that WithPrefix correctly set the flag, and add test. Also, add test for WithFromKey. fixes etcd-io#14056 Signed-off-by: Sahdev Zala <spzala@us.ibm.com>

…-#14182-upstream-release-3.5 Automated cherry pick of etcd-io#14182

The golang buildin package `flag` doesn't support `uint32` data type, so we need to support it via the `flag.Var`. Signed-off-by: Benjamin Wang <wachao@vmware.com>

…each client can open at a time Also refer to etcd-io#14169 (comment) Signed-off-by: Benjamin Wang <wachao@vmware.com>

Signed-off-by: Benjamin Wang <wachao@vmware.com>

[3.5] Support configuring `MaxConcurrentStreams` for http2

Signed-off-by: Jille Timmermans <jille@quis.cx>

[3.5] clientv3: Fix parsing of ETCD_CLIENT_DEBUG

Signed-off-by: Hitoshi Mitake <h.mitake@gmail.com>

server/auth: protect rangePermCache with a RW lock

This changes the default parent-based trace sampling rate from 100% to 0%. Due to the high QPS etcd can handle, having 100% trace sampling leads to very high resource usage. Defaulting to 0% means that only already-sampled traces will be sampled in etcd. Fixes etcd-io#14310 Signed-off-by: Mike Dame <mikedame@google.com>

Change default sampling rate from 100% to 0%

Signed-off-by: Benjamin Wang <wachao@vmware.com>

Upgrade grpc to 1.41.0; Run ./script/fix.sh to fix all related issue. Signed-off-by: Benjamin Wang <wachao@vmware.com>

Refer to etcd-io@a0bdfc4 Signed-off-by: Benjamin Wang <wachao@vmware.com>

Refer to etcd-io#14318 Signed-off-by: Benjamin Wang <wachao@vmware.com>

[3.5] etcdserver: bump OpenTelemetry to 1.0.1 and gRPC to 1.41.0

Streams are now closed after being used in the lessor `keepAliveOnce` method. This prevents the "failed to receive lease keepalive request from gRPC stream" message from being logged by the server after the context is cancelled by the client. Signed-off-by: Justin Kolberg <amd.prophet@gmail.com>

…stream [3.5] clientv3: close streams after use in lessor keepAliveOnce method

Only `net.TCPConn` supports `SetKeepAlive` and `SetKeepAlivePeriod` by default, so if you want to warp multiple layers of net.Listener, the `keepaliveListener` should be the one which is closest to the original `net.Listener` implementation, namely `TCPListener`. Also refer to etcd-io#14356 Signed-off-by: Benjamin Wang <wachao@vmware.com>

[3.5] Refactor the keepAliveListener and keepAliveConn

Signed-off-by: Vitalii Levitskii <vitalii@uber.com>

Problem: We pass grpc context down to applier in readonly serializable txn. This context can be cancelled for example due to timeout. This will trigger panic inside applyTxn Solution: Only panic for transactions with write operations fixes etcd-io#14110 main PR etcd-io#14149 Signed-off-by: Bogdan Kanivets <bkanivets@apple.com>

…oints-autosync Backport of pull/14354 to release-3.5

For a cluster with only one member, the raft always send identical unstable entries and committed entries to etcdserver, and etcd responds to the client once it finishes (actually partially) the applying workflow. When the client receives the response, it doesn't mean etcd has already successfully saved the data, including BoltDB and WAL, because: 1. etcd commits the boltDB transaction periodically instead of on each request; 2. etcd saves WAL entries in parallel with applying the committed entries. Accordingly, it may run into a situation of data loss when the etcd crashes immediately after responding to the client and before the boltDB and WAL successfully save the data to disk. Note that this issue can only happen for clusters with only one member. For clusters with multiple members, it isn't an issue, because etcd will not commit & apply the data before it being replicated to majority members. When the client receives the response, it means the data must have been applied. It further means the data must have been committed. Note: for clusters with multiple members, the raft will never send identical unstable entries and committed entries to etcdserver. Signed-off-by: Benjamin Wang <wachao@vmware.com>

…raft_3_5 [release-3.5] fix the potential data loss for clusters with only one member

Cherry pick etcd-io#14419 to 3.5. Signed-off-by: Benjamin Wang <wachao@vmware.com>

[3.5] Move consistent_index forward when executing alarmList operation

Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>

Get 100% coverage on InitialCheck and PeriodicCheck functions to avoid any mistakes. Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>

Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>

Makes it easier to test hash match between scheduleCompaction and HashByRev. Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>

Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>

… Defrag Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>

Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>

Fix corruption checks v3.5

This has been additionally verified by running the tests locally as a basic smoke test. GitHub Actions doesn't provide MacOS M1 (arm64) yet, so there's no good way to automate testing. Ran `TMPDIR=/tmp make test` locally. The `TMPDIR` bit is needed so there's no really long path used that breaks Unix socket setup in one of the tests. Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>

- permissions were incorrectly loaded on restarts. - etcd-io#14355 - Backport of etcd-io#14358 Signed-off-by: vivekpatani <9080894+vivekpatani@users.noreply.github.com>

[release-3.5] server,test: refresh cache on each NewAuthStore

[release-3.5] server/etcdmain: add build support for Apple M1

A WAL object was closed by defer, however the WAL was rewritten afterwards, so defer closed already closed WAL but not the new one. It caused a data race between writing file and cleaning up a temporary test directory, which led to a non-deterministic bug. Fixes etcd-io#14332 Signed-off-by: Vladimir Sokolov <vsvastey@gmail.com>

…-max-index-test-fix-3.5 [release-3.5] testing: fix TestOpenWithMaxIndex cleanup

Due to a duplicate call of clientConfigFromCmd, the move-leader command would fail with "conflicting environment variable is shadowed by corresponding command-line flag". Also in scenarios where no command-line flag was supplied. Signed-off-by: Thomas Jungblut <tjungblu@redhat.com>

etcdctl: allow move-leader to connect to multiple endpoints

[3.5] server: don't panic in readonly serializable txn

Signed-off-by: Benjamin Wang <wachao@vmware.com>

…er_20220913_3.5 [release-3.5] fix the flaky test TestV3AuthRestartMember

Commits on Feb 1, 2022

version: bump up to 3.5.2

serathius committed Feb 1, 2022

Configuration menu

View commit details

Copy full SHA for 99018a7

Browse repository at this point

Copy the full SHA

99018a7 View commit details

Browse the repository at this point in the history

Commits on Mar 13, 2022

Fix offline defrag

kkkkun committed Mar 13, 2022

Configuration menu

View commit details

Copy full SHA for 58374b8

Browse repository at this point

Copy the full SHA

58374b8 View commit details

Browse the repository at this point in the history

Commits on Apr 24, 2022

version: bump up to 3.5.4

serathius committed Apr 24, 2022

Configuration menu

View commit details

Copy full SHA for 08407ff

Browse repository at this point

Copy the full SHA

08407ff View commit details

Browse the repository at this point in the history

Commits on Sep 15, 2022

version: bump up to 3.5.5

serathius committed Sep 15, 2022

Configuration menu

View commit details

Copy full SHA for 19002cf

Browse repository at this point

Copy the full SHA

19002cf View commit details

Browse the repository at this point in the history

OCPBUGS-947: Rebase openshift/etcd 4.11 onto 3.5.5 #154

OCPBUGS-947: Rebase openshift/etcd 4.11 onto 3.5.5 #154

Commits on Jul 28, 2021

Commits on Jul 29, 2021

Commits on Aug 1, 2021

Commits on Aug 2, 2021

Commits on Aug 5, 2021

Commits on Aug 6, 2021

Commits on Aug 7, 2021

Commits on Aug 11, 2021

Commits on Aug 18, 2021

Commits on Aug 24, 2021

Commits on Sep 15, 2021

Commits on Sep 16, 2021

Commits on Sep 25, 2021

Commits on Sep 30, 2021

Commits on Oct 1, 2021

Commits on Oct 3, 2021

Commits on Oct 6, 2021

Commits on Oct 7, 2021

Commits on Oct 8, 2021

Commits on Oct 12, 2021

Commits on Oct 15, 2021

Commits on Oct 20, 2021

Commits on Nov 12, 2021

Commits on Nov 14, 2021

Commits on Nov 15, 2021

Commits on Nov 16, 2021

Commits on Nov 21, 2021

Commits on Nov 24, 2021

Commits on Dec 2, 2021

Commits on Dec 3, 2021

Commits on Dec 6, 2021

Commits on Dec 16, 2021

Commits on Dec 17, 2021

Commits on Dec 18, 2021

Commits on Dec 21, 2021

Commits on Jan 17, 2022

Commits on Feb 1, 2022

Commits on Feb 7, 2022

Commits on Feb 15, 2022

Commits on Feb 17, 2022

Commits on Feb 18, 2022

Commits on Feb 20, 2022

Commits on Feb 21, 2022

Commits on Feb 22, 2022

Commits on Feb 24, 2022

Commits on Feb 25, 2022

Commits on Mar 8, 2022

Commits on Mar 13, 2022

Commits on Mar 17, 2022

Commits on Mar 21, 2022

Commits on Mar 23, 2022

Commits on Mar 30, 2022

Commits on Mar 31, 2022

Commits on Apr 1, 2022

Commits on Apr 4, 2022

Commits on Apr 5, 2022

Commits on Apr 6, 2022

Commits on Apr 7, 2022

Commits on Apr 8, 2022

Commits on Apr 9, 2022

Commits on Apr 10, 2022

Commits on Apr 12, 2022

Commits on Apr 13, 2022

Commits on Apr 14, 2022

Commits on Apr 15, 2022

Commits on Apr 19, 2022

Commits on Apr 20, 2022

Commits on Apr 21, 2022

Commits on Apr 22, 2022

Commits on Apr 24, 2022

Commits on Apr 28, 2022

Commits on May 5, 2022

Commits on May 6, 2022

Commits on May 12, 2022

Commits on May 13, 2022

Commits on May 16, 2022

Commits on May 18, 2022

Commits on May 25, 2022