New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Blackhole failpoint in the proxy does not block all updates #17737
Comments
Can I attempt this issue :) @siyuanfoundation |
Of course :) Thank you for volunteering! |
Thank you! :) (Can you assign the issue to me ^_^?) |
cc @serathius @ahrtr |
Please note that we don't yet know the reason for blackholing working only on leader and not follower. I suspect that either blackhole proxy setup is wrong, blackhole is not doing its job or there is some traffic going around the proxy. To root cause the issue someone needs to use network analysis tools like tcpdump or wireshark to find how traffic is still going to the follower. |
@siyuanfoundation @serathius I think I managed to figure out why the blackhole is leaking traffic, and did a PoC to pass the e2e test that @siyuanfoundation provided with this commit. The main idea is that Raft nodes have 2 ways of communication - stream and pipeline. From what I see, the pipeline is indeed leaking traffic. Let's break the 2 communication means down in detail. StreamStream is covered by the proxy implementation since we have different port values for A proxy is used to forward traffic in between, so a node can see all the traffic sent from others. As the stream is initiated from the outside node to it (long-pooling), the traffic from external will pass through the proxy for sure. Thus, when we blackhole traffic, we can rewrite the data and set it to nil (and data with 0 length will be a dropped packet). PipelineFor pipeline, let's see the following arch drawing (sorry for my elementary-level drawing skill). Because the pipeline is initiated from the inside (node A) to the external nodes (node B and C), thus, it will bypass the proxy, like the red arrow is showing when node A is trying to create a pipeline and talk to node B and C. I believe this is the case because Since I am still very new to the codebase, I am not sure if my understanding is correct. But the PoC seems to work and it's based on the above thinking. If you guys think my understanding and explanation make sense somehow, please let me know how you would like to proceed with the fixes, as this PoC commit is clearly a hack (the blackhole flag is passed via file ... since I just want something that can work in a short time ^_^) Sidenote, on my machine, I sometimes run into one of the following 2 failures. But, if we make the following 2 changes in timing, I haven't seen the following 2 errors in 5 runs each. (
context deadline exceededNot sure what this is about yet.
revision mismatch after unblackholingI think without waiting for some time, the unblackholed node is still just coming back to the network and thus the catch up is not yet done.
|
@henrybear327 YES. The node dials to other members and read messages. The target follower is reaching out to other member's proxy. etcd/server/etcdserver/api/rafthttp/peer.go Line 230 in fa08f01
The member setups http server handle and setup a stream message writer. etcd/server/etcdserver/api/rafthttp/http.go Line 348 in fa08f01
The leader still can forward heartbeat and message to isolated follower. However, the outgoing connection from isolated follower to leader has been dropped. so, both the ReadIndex and proposal forward will be dropped from isolated follower. Maybe we can introduce new filter to drop any traffic from isolated follower. (Like L7 firewall 😂 ) |
Awesome findings @henrybear327 ! It is very well explained too! I think your findings are correct as shown by the codes @fuweid pointed to. |
Hey @siyuanfoundation it's before applying the 2 timing changes I would run into issues! |
I think we can still follow current design based on L4. However, we need to change func modifyTx(srcConn, dstConn net.Conn, data []byte) []byte {}
func modifyRx(srcConn, dstConn net.Conn, data []byte) []byte {}
// proxy pkg
type Server interface {
...
TxFilter(func(srcConn, dstConn net.Conn, data []byte) []byte)
RxFilter(func(srcConn, dstConn net.Conn, data []byte) []byte)
} For the customized filter, we use https://www.kernel.org/doc/Documentation/networking/proc_net_tcp.txt
For example, if we want to block the proxy -> isolated_member traffic, we can use customized modifiedRx: var isolatedMemberPid = x
func modifiedRx(srcConn, dstConn net.Conn, data []byte) []byte {
if ensureTupleFromPid(isolatedMemberPid, srcConn.RemoteAddr(), dstConn.RemoteAddr()) {
return nil
}
return data
} It works in my local. Since the e2e is using localhost, we can use inode to locate the target process. Yesterday, synced with @henrybear327 offline about using ping @serathius @ahrtr @aojea to seek review on this approach. |
The L4-level blocking sounds interesting! :) The main benefit would be we can block off the incoming and outgoing traffic for a specific pid without touching the etcdserver process's internal! :) The PoC that I have is using a different approach (but would require some internal failpoint injection): the main idea is that we are using channels to pass around messages the node is receiving and sending to/from the stream/pipeline, we can use failpoints to drop the messages coming in and out of the channels before they really get sent or processed by the node. This approach works, but there exists a small window where the message is already in the process of being sent that will not be covered by this failpoint channel blocking. Shouldn't be a big problem though since during e2e tests, the message size shouldn't be big. Based on @fuweid 's approach, I have another idea on L7-level blocking. https://github.com/henrybear327/etcd/commits/experiment/round_tripper/ Because both pipeline and stream are using We also need to work on the handler, too, to drop the incoming new connections. I think it's not as clean as the L4 blocking, but I think this is the code-level-wise blocking by leveraging the entry points of networking in/out functions, so we don't need external tools. |
So , this is not totally network partition, is just some of the flows get partitioned?
Before going to the implementations, it seems to me that this should happen at a higher level, if nodes are able to detect inconsistent network setups they should be able to announce or refuse to do any operations, no? |
The node will dial out to its peer using StreamReader and pipeline. Thus, the proxy is not in effect on those dialed out traffic. So yes, the current proxy only partially partition a node's traffic from the network. May I ask for more details for your second point? I don't quite get it :) Thank you! |
I'm not deep into the etcd internal details, but I expect the nodes to keep an state of their neighbours, and make decisions based on this state ... in this case it seems that there is some state that is "partial" , is not possible for the nodes to detect this and decide to stop operations? |
Hi @aojea yes. Each member has L4 proxy to forward peer's traffic to itself, described by the following flow.
I was thinking that if |
…io#17737 [Problem statement] The blackhole() leverages proxy to drop all the incoming and outgoing traffic passing through it, but the proxy doesn't properly block out all the communication channels for a given member, due to the fact that the member initiates connections from itself to others. [Overview of the proposed solutions] The proposed implementation here performs traffic blocking at L7 (application layer). There is another idea from fuweid performs blocking at L4, as mentioned in reference [1]. [Root cause] (Diagrams credit: fuweid) Let's assuming the following: - Current member ID (the member we would like to perform blackhole on): A - Other member ID: B, C In the e2e test set up, let's breakdown how A is able to communicate with its peers. For the case of incoming connections from B and C to A, the connections from B and C will be established through A-proxy. B -----> A-Proxy -----> A ^ | C For the case of establishing outgoing connections from A to B and C, the connections from A will be established through B-Proxy and C-Proxy, before reaching B and C, respectively. A -----> B-Proxy ----> B | +--------> C-Proxy ---> C As you can see, currently the `BlackholeTx` and `BlackholeRx` only blocks traffic between `X-Proxy <---> X`. Thus, only the externally-established incoming traffic will be block (B and C to A). [Implementation] For each member, it has 2 types of communication channels, namely stream and pipeline. Connections made by pipeline is not persisted, but for stream the connection is continuously used by the member for long-poling. In order to block the outgoing connections, we can hijack and drop the data from `RoundTrip` and `ServeHTTP`. When establishing a new connection, `RoundTrip` will be called. By using a failpoint, we can drop the data carried by `Body` in RequestBody on-demand. When accepting a new connection, `ServeHTTP` will be called. By using a failpoint, we can drop the data carried by `Body` in RequestBody, and drop the data written to `ResponseWriter` on-demand. [Discussion] The downside of this approach is that we are introducing a custom implementation of `RoundTrip`, as we are hijacking the connection. [Testing] make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 References: [1] etcd-io#17737 (comment) [2] https://github.com/etcd-io/etcd/pull/17790/files#diff-f01210a3082e25ff00682648f32122941a0c275b3926a8da37447589fe2ede1aR109
…io#17737 [Problem statement] The blackhole() leverages proxy to drop all the incoming and outgoing traffic passing through it, but the proxy doesn't properly block out all the communication channels for a given member, due to the fact that the member initiates connections from itself to others. [Overview of the proposed solutions] The proposed implementation here performs traffic blocking at L7 (application layer). There is another idea from fuweid performs blocking at L4, as mentioned in reference [1]. [Root cause] (Diagrams credit: fuweid) Let's assuming the following: - Current member ID (the member we would like to perform blackhole on): A - Other member ID: B, C In the e2e test set up, let's breakdown how A is able to communicate with its peers. For the case of incoming connections from B and C to A, the connections from B and C will be established through A-proxy. B -----> A-Proxy -----> A ^ | C For the case of establishing outgoing connections from A to B and C, the connections from A will be established through B-Proxy and C-Proxy, before reaching B and C, respectively. A -----> B-Proxy ----> B | +--------> C-Proxy ---> C As you can see, currently the `BlackholeTx` and `BlackholeRx` only blocks traffic between `X-Proxy <---> X`. Thus, only the externally-established incoming traffic will be block (B and C to A). [Implementation] For each member, it has 2 types of communication channels, namely stream and pipeline. Connections made by pipeline is not persisted, but for stream the connection is continuously used by the member for long-poling. In order to block the outgoing connections, we can hijack and drop the data from `RoundTrip` and `ServeHTTP`. When establishing a new connection, `RoundTrip` will be called. By using a failpoint, we can drop the data carried by `Body` in RequestBody on-demand. When accepting a new connection, `ServeHTTP` will be called. By using a failpoint, we can drop the data carried by `Body` in RequestBody, and drop the data written to `ResponseWriter` on-demand. If a connection is already opened (like a stream), because we hijacked the `Reader` and `Writer` interface, we can still drop traffic as desired. [Discussion] The downside of this approach is that we are introducing a custom implementation of `RoundTrip`, as we are hijacking the connection. [Testing] make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 References: [1] etcd-io#17737 (comment) [2] https://github.com/etcd-io/etcd/pull/17790/files#diff-f01210a3082e25ff00682648f32122941a0c275b3926a8da37447589fe2ede1aR109
…io#17737 [Problem statement] The blackhole() leverages proxy to drop all the incoming and outgoing traffic passing through it, but the proxy doesn't properly block out all the communication channels for a given member, due to the fact that the member initiates connections from itself to others. [Overview of the proposed solutions] The proposed implementation here performs traffic blocking at L7 (application layer). There is another idea from fuweid performs blocking at L4, as mentioned in reference [1]. [Root cause] (Diagrams credit: fuweid) Let's assuming the following: - Current member ID (the member we would like to perform blackhole on): A - Other member ID: B, C In the e2e test set up, let's breakdown how A is able to communicate with its peers. For the case of incoming connections from B and C to A, the connections from B and C will be established through A-proxy. B -----> A-Proxy -----> A ^ | C For the case of establishing outgoing connections from A to B and C, the connections from A will be established through B-Proxy and C-Proxy, before reaching B and C, respectively. A -----> B-Proxy ----> B | +--------> C-Proxy ---> C As you can see, currently the `BlackholeTx` and `BlackholeRx` only blocks traffic between `X-Proxy <---> X`. Thus, only the externally-established incoming traffic will be block (B and C to A). [Implementation] For each member, it has 2 types of communication channels, namely stream and pipeline. Connections made by pipeline is not persisted, but for stream the connection is continuously used by the member for long-poling. In order to block the outgoing connections, we can hijack and drop the data from `RoundTrip` and `ServeHTTP`. When establishing a new connection, `RoundTrip` will be called. By using a failpoint, we can drop the data carried by `Body` in RequestBody on-demand. When accepting a new connection, `ServeHTTP` will be called. By using a failpoint, we can drop the data carried by `Body` in RequestBody, and drop the data written to `ResponseWriter` on-demand. If a connection is already opened (like a stream), because we hijacked the `Reader` and `Writer` interface, we can still drop traffic as desired. [Discussion] The downside of this approach is that we are introducing a custom implementation of `RoundTrip`, as we are hijacking the connection. [Testing] make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 References: [1] etcd-io#17737 (comment) [2] https://github.com/etcd-io/etcd/pull/17790/files#diff-f01210a3082e25ff00682648f32122941a0c275b3926a8da37447589fe2ede1aR109 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
…io#17737 [Problem statement] The blackhole() leverages proxy to drop all the incoming and outgoing traffic passing through it, but the proxy doesn't properly block out all the communication channels for a given member, due to the fact that the member initiates connections from itself to others. [Overview of the proposed solutions] The proposed implementation here performs traffic blocking at L7 (application layer). There is another idea from fuweid performs blocking at L4, as mentioned in reference [1]. [Root cause] (Diagrams credit: fuweid) Let's assuming the following: - Current member ID (the member we would like to perform blackhole on): A - Other member ID: B, C In the e2e test set up, let's breakdown how A is able to communicate with its peers. For the case of incoming connections from B and C to A, the connections from B and C will be established through A-proxy. B -----> A-Proxy -----> A ^ | C For the case of establishing outgoing connections from A to B and C, the connections from A will be established through B-Proxy and C-Proxy, before reaching B and C, respectively. A -----> B-Proxy ----> B | +--------> C-Proxy ---> C As you can see, currently the `BlackholeTx` and `BlackholeRx` only blocks traffic between `X-Proxy <---> X`. Thus, only the externally-established incoming traffic will be block (B and C to A). [Implementation] For each member, it has 2 types of communication channels, namely stream and pipeline. Connections made by pipeline is not persisted, but for stream the connection is continuously used by the member for long-poling. In order to block the outgoing connections, we can hijack and drop the data from `RoundTrip` and `ServeHTTP`. When establishing a new connection, `RoundTrip` will be called. By using a failpoint, we can drop the data carried by `Body` in RequestBody on-demand. When accepting a new connection, `ServeHTTP` will be called. By using a failpoint, we can drop the data carried by `Body` in RequestBody, and drop the data written to `ResponseWriter` on-demand. If a connection is already opened (like a stream), because we hijacked the `Reader` and `Writer` interface, we can still drop traffic as desired. [Discussion] The downside of this approach is that we are introducing a custom implementation of `RoundTrip`, as we are hijacking the connection. [Testing] make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 References: [1] etcd-io#17737 (comment) [2] https://github.com/etcd-io/etcd/pull/17790/files#diff-f01210a3082e25ff00682648f32122941a0c275b3926a8da37447589fe2ede1aR109 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
…io#17737 [Problem statement] The blackhole() leverages proxy to drop all the incoming and outgoing traffic passing through it, but the proxy doesn't properly block out all the communication channels for a given member, due to the fact that the member initiates connections from itself to others. [Overview of the proposed solutions] The proposed implementation here performs traffic blocking at L7 (application layer). There is another idea from fuweid performs blocking at L4, as mentioned in reference [1]. [Root cause] (Diagrams credit: fuweid) Let's assuming the following: - Current member ID (the member we would like to perform blackhole on): A - Other member ID: B, C In the e2e test set up, let's breakdown how A is able to communicate with its peers. For the case of incoming connections from B and C to A, the connections from B and C will be established through A-proxy. B -----> A-Proxy -----> A ^ | C For the case of establishing outgoing connections from A to B and C, the connections from A will be established through B-Proxy and C-Proxy, before reaching B and C, respectively. A -----> B-Proxy ----> B | +--------> C-Proxy ---> C As you can see, currently the `BlackholeTx` and `BlackholeRx` only blocks traffic between `X-Proxy <---> X`. Thus, only the externally-established incoming traffic will be block (B and C to A). [Implementation] For each member, it has 2 types of communication channels, namely stream and pipeline. Connections made by pipeline is not persisted, but for stream the connection is continuously used by the member for long-poling. In order to block the outgoing connections, we can hijack and drop the data from `RoundTrip` and `ServeHTTP`. When establishing a new connection, `RoundTrip` will be called. By using a failpoint, we can drop the data carried by `Body` in RequestBody on-demand. When accepting a new connection, `ServeHTTP` will be called. By using a failpoint, we can drop the data carried by `Body` in RequestBody, and drop the data written to `ResponseWriter` on-demand. If a connection is already opened (like a stream), because we hijacked the `Reader` and `Writer` interface, we can still drop traffic as desired. [Discussion] The downside of this approach is that we are introducing a custom implementation of `RoundTrip`, as we are hijacking the connection. [Testing] make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 References: [1] etcd-io#17737 (comment) [2] https://github.com/etcd-io/etcd/pull/17790/files#diff-f01210a3082e25ff00682648f32122941a0c275b3926a8da37447589fe2ede1aR109 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Based on Fu Wei's idea discussed in the issue [1], we employ the blocking on L7 but without using external tools. [Problem] A peer will receive traffic from its peers and initiate connections to its peers (via stream and pipeline), thus, the current mechanism of only blocking traffic via proxy is incomplete, since the traffic initiated from the peers to others will be "leaking", since `blackholeTX` and `blackholeRX` only drop traffic coming in and out of the peer's proxy. [Solution - main idea] Let's first agree on the naming of the existing proxy as a "reverse proxy". We will introduce a "forward proxy", which will be proxying all the connection initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- B's SSL termination proxy - B's transparent proxy - B ^ newly introduced ^ in the original codebase ``` By adding this forward proxy, we can block all traffic coming in and out of a peer, without having to resort to external tools, such as iptables, and the blocking of traffic is complete. [Implementation] The main subtasks are: - Set up forward proxy - Enable/disable forward proxy [Test] make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 [Reference] [1] Issue etcd-io#17737 [2] Supersedes PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] Superseded PR (V2) etcd-io#17891
Based on Fu Wei's idea discussed in the issue [1], we employ the blocking on L7 but without using external tools. [Problem] A peer will (a) receive traffic from its peers and (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Solution - main idea] Let's first agree on the naming of the existing proxy as a "reverse proxy", since it sits in front of every peer and ingest the traffic into each of them. We introduce a "forward proxy" for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's forward proxy - B's transparent proxy - B ^ newly introduced ^ in the original codebase ``` By adding this forward proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `FORWARD_PROXY` - implement forward proxy by extending the existing proxy server code - implement enable/disable of the forward proxy in the e2e test [Test] make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 [Known issue] I run into `context deadline exceeded` sometimes. [References] [1] Issue etcd-io#17737 [2] Supersedes PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] Superseded PR (V2) etcd-io#17891
Based on Fu Wei's idea discussed in the issue [1], we employ the blocking on L7 but without using external tools. [Problem] A peer will (a) receive traffic from its peers and (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Solution - main idea] Let's first agree on the naming of the existing proxy as a "reverse proxy", since it sits in front of every peer and ingest the traffic into each of them. We introduce a "forward proxy" for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's forward proxy - B's transparent proxy - B ^ newly introduced ^ in the original codebase ``` By adding this forward proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `FORWARD_PROXY` - implement forward proxy by extending the existing proxy server code - implement enable/disable of the forward proxy in the e2e test [Test] make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 [Known issue] I run into `context deadline exceeded` sometimes. [References] [1] Issue etcd-io#17737 [2] Supersedes PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] Superseded PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Based on Fu Wei's idea discussed in the issue [1], we employ the blocking on L7 but without using external tools. [Problem] A peer will (a) receive traffic from its peers and (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Solution - main idea] Let's first agree on the naming of the existing proxy as a "reverse proxy", since it sits in front of every peer and ingest the traffic into each of them. We introduce a "forward proxy" for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's forward proxy - B's transparent proxy - B ^ newly introduced ^ in the original codebase ``` By adding this forward proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `FORWARD_PROXY` - implement forward proxy by extending the existing proxy server code - implement enable/disable of the forward proxy in the e2e test [Test] make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 [Known issue] I run into `context deadline exceeded` sometimes. [References] [1] Issue etcd-io#17737 [2] Supersedes PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] Superseded PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Based on Fu Wei's idea discussed in the issue [1], we employ the blocking on L7 but without using external tools. [Problem] A peer will (a) receive traffic from its peers and (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Solution - main idea] Let's first agree on the naming of the existing proxy as a "reverse proxy", since it sits in front of every peer and ingest the traffic into each of them. We introduce a "forward proxy" for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's forward proxy - B's reverse proxy - B ^ newly introduced ^ in the original codebase ``` By adding this forward proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `FORWARD_PROXY` - implement forward proxy by extending the existing proxy server code - implement enable/disable of the forward proxy in the e2e test [Test] make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 [Known issue] I run into `context deadline exceeded` sometimes. [References] [1] Issue etcd-io#17737 [2] Supersedes PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] Superseded PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Based on Fu Wei's idea discussed in the issue [1], we employ the blocking on L7 but without using external tools. [Problem] A peer will (a) receive traffic from its peers and (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Solution - main idea] Let's first agree on the naming of the existing proxy as a "reverse proxy", since it sits in front of every peer and ingest the traffic into each of them. We introduce a "forward proxy" for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's forward proxy - B's reverse proxy - B ^ newly introduced ^ in the original codebase ``` By adding this forward proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `FORWARD_PROXY` - implement forward proxy by extending the existing proxy server code - implement enable/disable of the forward proxy in the e2e test [Test] make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 [Known issue] I run into `context deadline exceeded` sometimes. [References] [1] Issue etcd-io#17737 [2] Supersedes PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] Superseded PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Based on Fu Wei's idea discussed in the issue [1], we employ the blocking on L7 but without using external tools. [Problem] A peer will (a) receive traffic from its peers and (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Solution - main idea] Let's first agree on the naming of the existing proxy as a "reverse proxy", since it sits in front of every peer and ingest the traffic into each of them. We introduce a "forward proxy" for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's forward proxy - B's reverse proxy - B ^ newly introduced ^ in the original codebase ``` By adding this forward proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `FORWARD_PROXY` - implement forward proxy by extending the existing proxy server code - implement enable/disable of the forward proxy in the e2e test [Test] make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 [Known issue] I run into `context deadline exceeded` sometimes. [References] [1] Issue etcd-io#17737 [2] Supersedes PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] Superseded PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Based on Fu Wei's idea discussed in the issue [1], we employ the blocking on L7 but without using external tools. [Problem] A peer will (a) receive traffic from its peers and (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Solution - main idea] Let's first agree on the naming of the existing proxy as a "reverse proxy", since it sits in front of every peer and ingest the traffic into each of them. We introduce a "forward proxy" for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's forward proxy - B's reverse proxy - B ^ newly introduced ^ in the original codebase ``` By adding this forward proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `FORWARD_PROXY` - implement forward proxy by extending the existing proxy server code - implement enable/disable of the forward proxy in the e2e test [Test] make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 [Known issue] I run into `context deadline exceeded` sometimes. [References] [1] Issue etcd-io#17737 [2] Supersedes PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] Superseded PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Based on Fu Wei's idea discussed in the issue [1], we employ the blocking on L7 but without using external tools. [Problem] A peer will (a) receive traffic from its peers and (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Solution - main idea] Let's first agree on the naming of the existing proxy as a "reverse proxy", since it sits in front of every peer and ingest the traffic into each of them. We introduce a "forward proxy" for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's forward proxy - B's reverse proxy - B ^ newly introduced ^ in the original codebase ``` By adding this forward proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `FORWARD_PROXY` - implement forward proxy by extending the existing proxy server code - implement enable/disable of the forward proxy in the e2e test [Test] make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 [Known issue] I run into `context deadline exceeded` sometimes. [References] [1] Issue etcd-io#17737 [2] Supersedes PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] Superseded PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Based on Fu Wei's idea discussed in the issue [1], we employ the blocking on L7 but without using external tools. [Problem] A peer will (a) receive traffic from its peers and (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Solution - main idea] Let's first agree on the naming of the existing proxy as a "reverse proxy", since it sits in front of every peer and ingest the traffic into each of them. We introduce a "forward proxy" for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's forward proxy - B's reverse proxy - B ^ newly introduced ^ in the original codebase ``` By adding this forward proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `FORWARD_PROXY` - implement forward proxy by extending the existing proxy server code - implement enable/disable of the forward proxy in the e2e test [Test] make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 [Known issue] I run into `context deadline exceeded` sometimes. [References] [1] Issue etcd-io#17737 [2] Supersedes PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] Superseded PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Based on Fu Wei's idea discussed in the issue [1], we employ the blocking on L7 but without using external tools. [Problem] A peer will (a) receive traffic from its peers and (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Solution - main idea] Let's first agree on the naming of the existing proxy as a "reverse proxy", since it sits in front of every peer and ingest the traffic into each of them. We introduce a "forward proxy" for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's forward proxy - B's reverse proxy - B ^ newly introduced ^ in the original codebase ``` By adding this forward proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `FORWARD_PROXY` - implement forward proxy by extending the existing proxy server code - implement enable/disable of the forward proxy in the e2e test [Test] make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 [Known issue] I run into `context deadline exceeded` sometimes. [References] [1] Issue etcd-io#17737 [2] Supersedes PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] Superseded PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Based on Fu Wei's idea discussed in the issue [1], we employ the blocking on L7 but without using external tools. [Problem] A peer will (a) receive traffic from its peers and (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Solution - main idea] Let's first agree on the naming of the existing proxy as a "reverse proxy", since it sits in front of every peer and ingest the traffic into each of them. We introduce a "forward proxy" for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's forward proxy - B's reverse proxy - B ^ newly introduced ^ in the original codebase ``` By adding this forward proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `FORWARD_PROXY` - implement forward proxy by extending the existing proxy server code - implement enable/disable of the forward proxy in the e2e test The end result is shown as the following diagram: ``` A -- A's forward proxy (connections initiated from A will be forward from this proxy) | ^ covers case (b) | --- A's reverse proxy (advertised to other peers where the connection should come in from) ^ covers case (a) ``` [Test] make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1 [Known issue] I run into `context deadline exceeded` sometimes. [References] [1] Issue etcd-io#17737 [2] Supersedes PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] Superseded PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Based on Fu Wei's idea discussed in the [issue](etcd-io#17737), we employ the blocking on L7 but without using external tools. A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. We introduce an "HTTP proxy" for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's HTTP proxy ----- B's (currently existing) proxy - B ^ newly introduced ^ in the original codebase ``` By adding this HTTP proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. The main subtasks are - set up an environment variable `FORWARD_PROXY` - implement HTTP proxy by extending the existing proxy server code - implement enable/disable of the HTTP proxy in the e2e test The result is that for every peer, we will have the arch like this ``` A -- A's HTTP proxy (connections initiated from A will be forwarded from this proxy) | ^ covers case (b) | --- A's (currently existing) proxy (advertised to other peers where the connection should come in from) ^ covers case (a) ``` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - I run into `context deadline exceeded` sometimes ``` etcd_mix_versions_test.go:175: Error Trace: /Users/henrybear327/go/src/etcd/tests/e2e/etcd_mix_versions_test.go:175 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:75 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:31 Error: Received unexpected error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE Test: TestBlackholeByMockingPartitionLeader Messages: failed to put "key-0", error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE ``` [References] [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Thanks to Fu Wei for the input regarding etcd-io#17938. [Problem] A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Main idea] We introduce 1 shared "HTTP proxy" for all peers. All peers will be proxying all the connections through it. The modified architecture will look something like this: ``` A -- shared HTTP proxy ----- B ^ newly introduced ``` By adding this HTTP proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `FORWARD_PROXY`, because go will not parse HTTP_PROXY and HTTPS_PROXY that is using localhost or 127.0.0.1, regardless if the port is present or not - implement the shared HTTP proxy by extending the existing proxy server code - remove existing proxy setup (the per-peer proxy) - implement enable/disable of the HTTP proxy in the e2e test [Testing] - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` [References] [1] Tracking issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 [4] PR (V3) etcd-io#17938 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Thanks to Fu Wei for the input regarding etcd-io#17938. [Problem] A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Main idea] We introduce 1 shared HTTP proxy for all peers. All peers will be proxying all the connections through it. The modified architecture will look something like this: ``` A -- shared HTTP proxy ----- B ^ newly introduced ``` By adding this HTTP proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `FORWARD_PROXY`, because go will not parse HTTP_PROXY and HTTPS_PROXY that is using localhost or 127.0.0.1, regardless if the port is present or not - implement the shared HTTP proxy by extending the existing proxy server code (we need to be able to identify the source sender) - remove existing proxy setup (the per-peer proxy) - implement enable/disable of the HTTP proxy in the e2e test [Testing] - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` [References] [1] Tracking issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 [4] PR (V3) etcd-io#17938 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Based on Fu Wei's idea discussed in the [issue](etcd-io#17737), we employ the blocking on L7 but without using external tools. A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. We introduce an "HTTP proxy" for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's HTTP proxy ----- B's (currently existing) proxy - B ^ newly introduced ^ in the original codebase ``` By adding this HTTP proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. The main subtasks are - set up an environment variable `FORWARD_PROXY` - implement HTTP proxy by extending the existing proxy server code - implement enable/disable of the HTTP proxy in the e2e test The result is that for every peer, we will have the arch like this ``` A -- A's HTTP proxy (connections initiated from A will be forwarded from this proxy) | ^ covers case (b) | --- A's (currently existing) proxy (advertised to other peers where the connection should come in from) ^ covers case (a) ``` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - I run into `context deadline exceeded` sometimes ``` etcd_mix_versions_test.go:175: Error Trace: /Users/henrybear327/go/src/etcd/tests/e2e/etcd_mix_versions_test.go:175 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:75 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:31 Error: Received unexpected error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE Test: TestBlackholeByMockingPartitionLeader Messages: failed to put "key-0", error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE ``` [References] [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Based on Fu Wei's idea discussed in the [issue](etcd-io#17737), we employ the blocking on L7 but without using external tools. A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. We introduce an "HTTP proxy" for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's HTTP proxy ----- B's (currently existing) proxy - B ^ newly introduced ^ in the original codebase ``` By adding this HTTP proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. The main subtasks are - set up an environment variable `FORWARD_PROXY` - implement HTTP proxy by extending the existing proxy server code - implement enable/disable of the HTTP proxy in the e2e test The result is that for every peer, we will have the arch like this ``` A -- A's HTTP proxy (connections initiated from A will be forwarded from this proxy) | ^ covers case (b) | --- A's (currently existing) proxy (advertised to other peers where the connection should come in from) ^ covers case (a) ``` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - I run into `context deadline exceeded` sometimes ``` etcd_mix_versions_test.go:175: Error Trace: /Users/henrybear327/go/src/etcd/tests/e2e/etcd_mix_versions_test.go:175 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:75 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:31 Error: Received unexpected error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE Test: TestBlackholeByMockingPartitionLeader Messages: failed to put "key-0", error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE ``` [References] [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Thanks to Fu Wei for the input regarding etcd-io#17938. [Problem] A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Main idea] We introduce 1 shared HTTP proxy for all peers. All peers will be proxying all the connections through it. The modified architecture will look something like this: ``` A -- shared HTTP proxy ----- B ^ newly introduced ``` By adding this HTTP proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `FORWARD_PROXY`, because go will not parse HTTP_PROXY and HTTPS_PROXY that is using localhost or 127.0.0.1, regardless if the port is present or not - implement the shared HTTP proxy by extending the existing proxy server code (we need to be able to identify the source sender) - remove existing proxy setup (the per-peer proxy) - implement enable/disable of the HTTP proxy in the e2e test [Testing] - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` [References] [1] Tracking issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 [4] PR (V3) etcd-io#17938 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Based on Fu Wei's idea discussed in the [issue](etcd-io#17737), we employ the blocking on L7 but without using external tools. A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. We introduce an "HTTP proxy" for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's HTTP proxy ----- B's (currently existing) proxy - B ^ newly introduced ^ in the original codebase ``` By adding this HTTP proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. The main subtasks are - set up an environment variable `FORWARD_PROXY` - implement HTTP proxy by extending the existing proxy server code - implement enable/disable of the HTTP proxy in the e2e test The result is that for every peer, we will have the arch like this ``` A -- A's HTTP proxy (connections initiated from A will be forwarded from this proxy) | ^ covers case (b) | --- A's (currently existing) proxy (advertised to other peers where the connection should come in from) ^ covers case (a) ``` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - I run into `context deadline exceeded` sometimes ``` etcd_mix_versions_test.go:175: Error Trace: /Users/henrybear327/go/src/etcd/tests/e2e/etcd_mix_versions_test.go:175 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:75 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:31 Error: Received unexpected error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE Test: TestBlackholeByMockingPartitionLeader Messages: failed to put "key-0", error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE ``` [References] [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Based on Fu Wei's idea discussed in the [issue](etcd-io#17737), we employ the blocking on L7 but without using external tools. [Background] A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Proposed solution] We introduce an forward proxy for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's forward proxy ----- B's reverse proxy - B ^ newly introduced ^ in the original codebase (renamed) ``` By adding this forward proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `E2E_TEST_FORWARD_PROXY_IP` - implement forward proxy by extending the existing proxy server code - implement enable/disable of the forward proxy in the e2e test The result is that for every peer, we will have the arch like this ``` A -- A's forward proxy (connections initiated from A will be forwarded from this proxy) | ^ covers case (b) | --- A's (currently existing) reverse proxy (advertised to other peers where the connection should come in from) ^ covers case (a) ``` [Testing] - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` [Issues] - I run into `context deadline exceeded` sometimes ``` etcd_mix_versions_test.go:175: Error Trace: /Users/henrybear327/go/src/etcd/tests/e2e/etcd_mix_versions_test.go:175 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:75 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:31 Error: Received unexpected error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE Test: TestBlackholeByMockingPartitionLeader Messages: failed to put "key-0", error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE ``` [References] [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Based on Fu Wei's idea discussed in the [issue](etcd-io#17737), we employ the blocking on L7 but without using external tools. [Background] A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Proposed solution] We introduce an forward proxy for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's forward proxy ----- B's reverse proxy - B ^ newly introduced ^ in the original codebase (renamed) ``` By adding this forward proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `E2E_TEST_FORWARD_PROXY_IP` - implement forward proxy by extending the existing proxy server code - implement enable/disable of the forward proxy in the e2e test The result is that for every peer, we will have the arch like this ``` A -- A's forward proxy (connections initiated from A will be forwarded from this proxy) | ^ covers case (b) | --- A's (currently existing) reverse proxy (advertised to other peers where the connection should come in from) ^ covers case (a) ``` [Testing] - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` [Issues] - I run into `context deadline exceeded` sometimes ``` etcd_mix_versions_test.go:175: Error Trace: /Users/henrybear327/go/src/etcd/tests/e2e/etcd_mix_versions_test.go:175 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:75 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:31 Error: Received unexpected error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE Test: TestBlackholeByMockingPartitionLeader Messages: failed to put "key-0", error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE ``` [References] [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Based on Fu Wei's idea discussed in the [issue](etcd-io#17737), we employ the blocking on L7 but without using external tools. [Background] A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Proposed solution] We introduce an forward proxy for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's forward proxy ----- B's reverse proxy - B ^ newly introduced ^ in the original codebase (renamed) ``` By adding this forward proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. [Testing] - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` [Issues] - I run into `context deadline exceeded` sometimes ``` etcd_mix_versions_test.go:175: Error Trace: /Users/henrybear327/go/src/etcd/tests/e2e/etcd_mix_versions_test.go:175 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:75 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:31 Error: Received unexpected error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE Test: TestBlackholeByMockingPartitionLeader Messages: failed to put "key-0", error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE ``` [References] [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com> It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `E2E_TEST_FORWARD_PROXY_IP` - implement forward proxy by extending the existing proxy server code - implement enable/disable of the forward proxy in the e2e test The result is that for every peer, we will have the arch like this ``` A -- A's forward proxy (connections initiated from A will be forwarded from this proxy) | ^ covers case (b) | --- A's (currently existing) reverse proxy (advertised to other peers where the connection should come in from) ^ covers case (a) ```
Based on Fu Wei's idea discussed in the [issue](etcd-io#17737), we employ the blocking on L7 but without using external tools. [Background] A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Proposed solution] We introduce an forward proxy for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's forward proxy ----- B's reverse proxy - B ^ newly introduced ^ in the original codebase (renamed) ``` By adding this forward proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. [Testing] - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` [Issues] - I run into `context deadline exceeded` sometimes ``` etcd_mix_versions_test.go:175: Error Trace: /Users/henrybear327/go/src/etcd/tests/e2e/etcd_mix_versions_test.go:175 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:75 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:31 Error: Received unexpected error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE Test: TestBlackholeByMockingPartitionLeader Messages: failed to put "key-0", error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE ``` [References] [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com> It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `E2E_TEST_FORWARD_PROXY_IP` - implement forward proxy by extending the existing proxy server code - implement enable/disable of the forward proxy in the e2e test The result is that for every peer, we will have the arch like this ``` A -- A's forward proxy (connections initiated from A will be forwarded from this proxy) | ^ covers case (b) | --- A's (currently existing) reverse proxy (advertised to other peers where the connection should come in from) ^ covers case (a) ```
Bug report criteria
What happened?
When mocking a network partition in e2e test with the
proxy.BlackholeTx()
andproxy.BlackholeRx()
, the partitioned follower node can still received all the write updates happening during that blackhole period.When the partitioned node was the original leader, new write updates are not applied to the partitioned node.
This bug makes the reliability of existing tests depending on this failpoint questionable.
What did you expect to happen?
The blackhole failpoint should drop all packets sent to the partitioned node, and it should not receive any write updates happening during that blackhole period.
How can we reproduce it (as minimally and precisely as possible)?
#17736
Anything else we need to know?
No response
Etcd version (please run commands below)
Etcd configuration (command line flags or environment variables)
paste your configuration here
Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)
Relevant log output
No response
The text was updated successfully, but these errors were encountered: