[RFC] Attempt to make Blockhole blocking on L7 #17891
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Hi @henrybear327. Thanks for your PR. I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
Thanks to @fuweid for suggesting to look into #15271 I applied the PR to the robustness test We can see that the revision log is the following: Compared to before patching this PR, and the robustness test failed on my machine, but not with the So I think the PR is working in a sense, as the blackholedMemberRevision is fixed at a version. |
henrybear327
force-pushed
the
fix/e2e_blackhole_proxy_review
branch
from
236d14b
to
60c5240
Compare
|
Can you please take a look when you have time, and see if this PR idea is worth refining into a reviewable PR :) I have tested it on the robustness test, and attached the log and findings! |
|
I think using a separate proxy type makes sense, and the remaining 3 leaking traffic should not really affect p2p communication much. |
Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Shorten the wait time for the open connection to expire to 5s Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Based on Fu Wei's idea, we employ blocking on L7 but without using
external tools.
[Main ideas]
The main idea is
1) utilize the `X-PeerURLs` field from the header, as we know all
traffic to peer will contain this header field.
2) As we also know that all nodes will create direct connections with their
peers, so the traffic blocking will have to happen at all nodes'
proxies, contrary to the current design, where only the proxy of the
peer is being blackholed.
Based on the main ideas, we introduce a SSL termination proxy so we can
obtain the `X-PeerURLs` field from the header.
[Issues]
There are 2 known issues with this approach
1) still leaking some traffic. But the leaked traffic (as discussed
later won't affect the blackhole idea that we would like to achieve (as
stream and pipeline traffic between raft nodes are now properly terminated)
2) we would need to employ SSL termination proxy, which might lead to a
small performance hit
For 1), as this way of blocking (by utilizing `X-PeerURLs` from the
header) will miss certain types of traffic to certain endpoints, as
the traffic to some endpoints doesn't have the `X-PeerURLs` field.
Currently, here are the known ones: /members, /version, and /raft/probing.
As you can see from the log, its header doesn't contain the `X-PeerURLs`
field, but only the following fields:
- map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]] /members
- map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]] /version
- map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]] /raft/probing
For 2) in order to read out `X-PeerURLs` from the header, we need to
terminate the SSL connection, as we can't drop cleartext traffic
(ref [1]). Thus, a new option `e2e.WithSSLTerminationProxy(true)`
is introduced, which will change the network flow into
```
A -- B's SSL termination proxy - B's transparent proxy - B
^ newly introduced ^ in the original codebase
```
[Known improvements required before turning RFC into PR]
The prototype needs to be further improved for code review after
fixing the following issues:
- blocking only RX or TX traffic (as currently a call to `blackholeTX`
or `blackholeRX` will black both TX and RX traffic instead of just
the specified one.
- slowness when performing test cleanup (I think this is related to the
SSL timeout setting, but I haven't looked into it yet)
- coding style improvements
References:
[1] etcd-io#15595
Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
henrybear327
force-pushed
the
fix/e2e_blackhole_proxy_review
branch
from
60c5240
to
930dde5
Compare
fuweid
reviewed
May 4, 2024
|
A different approach is used to supersede this PR, thus closing this one. |
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 5, 2024
Based on Fu Wei's idea discussed in the issue [1], we employ the
blocking on L7 but without using external tools.
[Problem]
A peer will receive traffic from its peers and initiate connections to
its peers (via stream and pipeline), thus, the current mechanism of
only blocking traffic via proxy is incomplete, since the traffic
initiated from the peers to others will be "leaking", since `blackholeTX`
and `blackholeRX` only drop traffic coming in and out of the peer's
proxy.
[Solution - main idea]
Let's first agree on the naming of the existing proxy as a
"reverse proxy".
We will introduce a "forward proxy", which will be proxying all the
connection initiated from a peer to its peers.
The modified architecture will look something like this:
```
A -- B's SSL termination proxy - B's transparent proxy - B
^ newly introduced ^ in the original codebase
```
By adding this forward proxy, we can block all traffic coming in and out
of a peer, without having to resort to external tools, such as iptables,
and the blocking of traffic is complete.
[Implementation]
The main subtasks are:
- Set up forward proxy
- Enable/disable forward proxy
[Test]
make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1
make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1
[Reference]
[1] Issue etcd-io#17737
[2] Supersedes PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] Superseded PR (V2) etcd-io#17891
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 5, 2024
Based on Fu Wei's idea discussed in the issue [1], we employ the
blocking on L7 but without using external tools.
[Problem]
A peer will (a) receive traffic from its peers and (b) initiate
connections to its peers (via stream and pipeline).
Thus, the current mechanism of only blocking peer traffic via the peer's
existing proxy is insufficient, since only scenario (a) is handled, and
scenario (b) is not blocked at all.
[Solution - main idea]
Let's first agree on the naming of the existing proxy as a
"reverse proxy", since it sits in front of every peer and ingest the
traffic into each of them.
We introduce a "forward proxy" for each peer, which will be proxying all
the connections initiated from a peer to its peers.
The modified architecture will look something like this:
```
A -- A's forward proxy - B's transparent proxy - B
^ newly introduced ^ in the original codebase
```
By adding this forward proxy, we can block all in and out traffic that
is initiated from a peer to others, without having to resort to external
tools, such as iptables. It's verified that the blocking of traffic is
complete, compared to previous solutions [2][3].
[Implementation]
The main subtasks are
- set up an environment variable `FORWARD_PROXY`
- implement forward proxy by extending the existing proxy server code
- implement enable/disable of the forward proxy in the e2e test
[Test]
make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1
make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1
[Known issue]
I run into `context deadline exceeded` sometimes.
[References]
[1] Issue etcd-io#17737
[2] Supersedes PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] Superseded PR (V2) etcd-io#17891
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 5, 2024
Based on Fu Wei's idea discussed in the issue [1], we employ the
blocking on L7 but without using external tools.
[Problem]
A peer will (a) receive traffic from its peers and (b) initiate
connections to its peers (via stream and pipeline).
Thus, the current mechanism of only blocking peer traffic via the peer's
existing proxy is insufficient, since only scenario (a) is handled, and
scenario (b) is not blocked at all.
[Solution - main idea]
Let's first agree on the naming of the existing proxy as a
"reverse proxy", since it sits in front of every peer and ingest the
traffic into each of them.
We introduce a "forward proxy" for each peer, which will be proxying all
the connections initiated from a peer to its peers.
The modified architecture will look something like this:
```
A -- A's forward proxy - B's transparent proxy - B
^ newly introduced ^ in the original codebase
```
By adding this forward proxy, we can block all in and out traffic that
is initiated from a peer to others, without having to resort to external
tools, such as iptables. It's verified that the blocking of traffic is
complete, compared to previous solutions [2][3].
[Implementation]
The main subtasks are
- set up an environment variable `FORWARD_PROXY`
- implement forward proxy by extending the existing proxy server code
- implement enable/disable of the forward proxy in the e2e test
[Test]
make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1
make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1
[Known issue]
I run into `context deadline exceeded` sometimes.
[References]
[1] Issue etcd-io#17737
[2] Supersedes PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] Superseded PR (V2) etcd-io#17891
Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 5, 2024
Based on Fu Wei's idea discussed in the issue [1], we employ the
blocking on L7 but without using external tools.
[Problem]
A peer will (a) receive traffic from its peers and (b) initiate
connections to its peers (via stream and pipeline).
Thus, the current mechanism of only blocking peer traffic via the peer's
existing proxy is insufficient, since only scenario (a) is handled, and
scenario (b) is not blocked at all.
[Solution - main idea]
Let's first agree on the naming of the existing proxy as a
"reverse proxy", since it sits in front of every peer and ingest the
traffic into each of them.
We introduce a "forward proxy" for each peer, which will be proxying all
the connections initiated from a peer to its peers.
The modified architecture will look something like this:
```
A -- A's forward proxy - B's transparent proxy - B
^ newly introduced ^ in the original codebase
```
By adding this forward proxy, we can block all in and out traffic that
is initiated from a peer to others, without having to resort to external
tools, such as iptables. It's verified that the blocking of traffic is
complete, compared to previous solutions [2][3].
[Implementation]
The main subtasks are
- set up an environment variable `FORWARD_PROXY`
- implement forward proxy by extending the existing proxy server code
- implement enable/disable of the forward proxy in the e2e test
[Test]
make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1
make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1
[Known issue]
I run into `context deadline exceeded` sometimes.
[References]
[1] Issue etcd-io#17737
[2] Supersedes PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] Superseded PR (V2) etcd-io#17891
Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 5, 2024
Based on Fu Wei's idea discussed in the issue [1], we employ the
blocking on L7 but without using external tools.
[Problem]
A peer will (a) receive traffic from its peers and (b) initiate
connections to its peers (via stream and pipeline).
Thus, the current mechanism of only blocking peer traffic via the peer's
existing proxy is insufficient, since only scenario (a) is handled, and
scenario (b) is not blocked at all.
[Solution - main idea]
Let's first agree on the naming of the existing proxy as a
"reverse proxy", since it sits in front of every peer and ingest the
traffic into each of them.
We introduce a "forward proxy" for each peer, which will be proxying all
the connections initiated from a peer to its peers.
The modified architecture will look something like this:
```
A -- A's forward proxy - B's reverse proxy - B
^ newly introduced ^ in the original codebase
```
By adding this forward proxy, we can block all in and out traffic that
is initiated from a peer to others, without having to resort to external
tools, such as iptables. It's verified that the blocking of traffic is
complete, compared to previous solutions [2][3].
[Implementation]
The main subtasks are
- set up an environment variable `FORWARD_PROXY`
- implement forward proxy by extending the existing proxy server code
- implement enable/disable of the forward proxy in the e2e test
[Test]
make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1
make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1
[Known issue]
I run into `context deadline exceeded` sometimes.
[References]
[1] Issue etcd-io#17737
[2] Supersedes PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] Superseded PR (V2) etcd-io#17891
Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 5, 2024
Based on Fu Wei's idea discussed in the issue [1], we employ the
blocking on L7 but without using external tools.
[Problem]
A peer will (a) receive traffic from its peers and (b) initiate
connections to its peers (via stream and pipeline).
Thus, the current mechanism of only blocking peer traffic via the peer's
existing proxy is insufficient, since only scenario (a) is handled, and
scenario (b) is not blocked at all.
[Solution - main idea]
Let's first agree on the naming of the existing proxy as a
"reverse proxy", since it sits in front of every peer and ingest the
traffic into each of them.
We introduce a "forward proxy" for each peer, which will be proxying all
the connections initiated from a peer to its peers.
The modified architecture will look something like this:
```
A -- A's forward proxy - B's reverse proxy - B
^ newly introduced ^ in the original codebase
```
By adding this forward proxy, we can block all in and out traffic that
is initiated from a peer to others, without having to resort to external
tools, such as iptables. It's verified that the blocking of traffic is
complete, compared to previous solutions [2][3].
[Implementation]
The main subtasks are
- set up an environment variable `FORWARD_PROXY`
- implement forward proxy by extending the existing proxy server code
- implement enable/disable of the forward proxy in the e2e test
[Test]
make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1
make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1
[Known issue]
I run into `context deadline exceeded` sometimes.
[References]
[1] Issue etcd-io#17737
[2] Supersedes PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] Superseded PR (V2) etcd-io#17891
Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 5, 2024
Based on Fu Wei's idea discussed in the issue [1], we employ the
blocking on L7 but without using external tools.
[Problem]
A peer will (a) receive traffic from its peers and (b) initiate
connections to its peers (via stream and pipeline).
Thus, the current mechanism of only blocking peer traffic via the peer's
existing proxy is insufficient, since only scenario (a) is handled, and
scenario (b) is not blocked at all.
[Solution - main idea]
Let's first agree on the naming of the existing proxy as a
"reverse proxy", since it sits in front of every peer and ingest the
traffic into each of them.
We introduce a "forward proxy" for each peer, which will be proxying all
the connections initiated from a peer to its peers.
The modified architecture will look something like this:
```
A -- A's forward proxy - B's reverse proxy - B
^ newly introduced ^ in the original codebase
```
By adding this forward proxy, we can block all in and out traffic that
is initiated from a peer to others, without having to resort to external
tools, such as iptables. It's verified that the blocking of traffic is
complete, compared to previous solutions [2][3].
[Implementation]
The main subtasks are
- set up an environment variable `FORWARD_PROXY`
- implement forward proxy by extending the existing proxy server code
- implement enable/disable of the forward proxy in the e2e test
[Test]
make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1
make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1
[Known issue]
I run into `context deadline exceeded` sometimes.
[References]
[1] Issue etcd-io#17737
[2] Supersedes PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] Superseded PR (V2) etcd-io#17891
Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 5, 2024
Based on Fu Wei's idea discussed in the issue [1], we employ the
blocking on L7 but without using external tools.
[Problem]
A peer will (a) receive traffic from its peers and (b) initiate
connections to its peers (via stream and pipeline).
Thus, the current mechanism of only blocking peer traffic via the peer's
existing proxy is insufficient, since only scenario (a) is handled, and
scenario (b) is not blocked at all.
[Solution - main idea]
Let's first agree on the naming of the existing proxy as a
"reverse proxy", since it sits in front of every peer and ingest the
traffic into each of them.
We introduce a "forward proxy" for each peer, which will be proxying all
the connections initiated from a peer to its peers.
The modified architecture will look something like this:
```
A -- A's forward proxy - B's reverse proxy - B
^ newly introduced ^ in the original codebase
```
By adding this forward proxy, we can block all in and out traffic that
is initiated from a peer to others, without having to resort to external
tools, such as iptables. It's verified that the blocking of traffic is
complete, compared to previous solutions [2][3].
[Implementation]
The main subtasks are
- set up an environment variable `FORWARD_PROXY`
- implement forward proxy by extending the existing proxy server code
- implement enable/disable of the forward proxy in the e2e test
[Test]
make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1
make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1
[Known issue]
I run into `context deadline exceeded` sometimes.
[References]
[1] Issue etcd-io#17737
[2] Supersedes PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] Superseded PR (V2) etcd-io#17891
Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 5, 2024
Based on Fu Wei's idea discussed in the issue [1], we employ the
blocking on L7 but without using external tools.
[Problem]
A peer will (a) receive traffic from its peers and (b) initiate
connections to its peers (via stream and pipeline).
Thus, the current mechanism of only blocking peer traffic via the peer's
existing proxy is insufficient, since only scenario (a) is handled, and
scenario (b) is not blocked at all.
[Solution - main idea]
Let's first agree on the naming of the existing proxy as a
"reverse proxy", since it sits in front of every peer and ingest the
traffic into each of them.
We introduce a "forward proxy" for each peer, which will be proxying all
the connections initiated from a peer to its peers.
The modified architecture will look something like this:
```
A -- A's forward proxy - B's reverse proxy - B
^ newly introduced ^ in the original codebase
```
By adding this forward proxy, we can block all in and out traffic that
is initiated from a peer to others, without having to resort to external
tools, such as iptables. It's verified that the blocking of traffic is
complete, compared to previous solutions [2][3].
[Implementation]
The main subtasks are
- set up an environment variable `FORWARD_PROXY`
- implement forward proxy by extending the existing proxy server code
- implement enable/disable of the forward proxy in the e2e test
[Test]
make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1
make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1
[Known issue]
I run into `context deadline exceeded` sometimes.
[References]
[1] Issue etcd-io#17737
[2] Supersedes PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] Superseded PR (V2) etcd-io#17891
Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 5, 2024
Based on Fu Wei's idea discussed in the issue [1], we employ the
blocking on L7 but without using external tools.
[Problem]
A peer will (a) receive traffic from its peers and (b) initiate
connections to its peers (via stream and pipeline).
Thus, the current mechanism of only blocking peer traffic via the peer's
existing proxy is insufficient, since only scenario (a) is handled, and
scenario (b) is not blocked at all.
[Solution - main idea]
Let's first agree on the naming of the existing proxy as a
"reverse proxy", since it sits in front of every peer and ingest the
traffic into each of them.
We introduce a "forward proxy" for each peer, which will be proxying all
the connections initiated from a peer to its peers.
The modified architecture will look something like this:
```
A -- A's forward proxy - B's reverse proxy - B
^ newly introduced ^ in the original codebase
```
By adding this forward proxy, we can block all in and out traffic that
is initiated from a peer to others, without having to resort to external
tools, such as iptables. It's verified that the blocking of traffic is
complete, compared to previous solutions [2][3].
[Implementation]
The main subtasks are
- set up an environment variable `FORWARD_PROXY`
- implement forward proxy by extending the existing proxy server code
- implement enable/disable of the forward proxy in the e2e test
[Test]
make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1
make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1
[Known issue]
I run into `context deadline exceeded` sometimes.
[References]
[1] Issue etcd-io#17737
[2] Supersedes PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] Superseded PR (V2) etcd-io#17891
Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 5, 2024
Based on Fu Wei's idea discussed in the issue [1], we employ the
blocking on L7 but without using external tools.
[Problem]
A peer will (a) receive traffic from its peers and (b) initiate
connections to its peers (via stream and pipeline).
Thus, the current mechanism of only blocking peer traffic via the peer's
existing proxy is insufficient, since only scenario (a) is handled, and
scenario (b) is not blocked at all.
[Solution - main idea]
Let's first agree on the naming of the existing proxy as a
"reverse proxy", since it sits in front of every peer and ingest the
traffic into each of them.
We introduce a "forward proxy" for each peer, which will be proxying all
the connections initiated from a peer to its peers.
The modified architecture will look something like this:
```
A -- A's forward proxy - B's reverse proxy - B
^ newly introduced ^ in the original codebase
```
By adding this forward proxy, we can block all in and out traffic that
is initiated from a peer to others, without having to resort to external
tools, such as iptables. It's verified that the blocking of traffic is
complete, compared to previous solutions [2][3].
[Implementation]
The main subtasks are
- set up an environment variable `FORWARD_PROXY`
- implement forward proxy by extending the existing proxy server code
- implement enable/disable of the forward proxy in the e2e test
[Test]
make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1
make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1
[Known issue]
I run into `context deadline exceeded` sometimes.
[References]
[1] Issue etcd-io#17737
[2] Supersedes PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] Superseded PR (V2) etcd-io#17891
Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 6, 2024
Based on Fu Wei's idea discussed in the issue [1], we employ the
blocking on L7 but without using external tools.
[Problem]
A peer will (a) receive traffic from its peers and (b) initiate
connections to its peers (via stream and pipeline).
Thus, the current mechanism of only blocking peer traffic via the peer's
existing proxy is insufficient, since only scenario (a) is handled, and
scenario (b) is not blocked at all.
[Solution - main idea]
Let's first agree on the naming of the existing proxy as a
"reverse proxy", since it sits in front of every peer and ingest the
traffic into each of them.
We introduce a "forward proxy" for each peer, which will be proxying all
the connections initiated from a peer to its peers.
The modified architecture will look something like this:
```
A -- A's forward proxy - B's reverse proxy - B
^ newly introduced ^ in the original codebase
```
By adding this forward proxy, we can block all in and out traffic that
is initiated from a peer to others, without having to resort to external
tools, such as iptables. It's verified that the blocking of traffic is
complete, compared to previous solutions [2][3].
[Implementation]
The main subtasks are
- set up an environment variable `FORWARD_PROXY`
- implement forward proxy by extending the existing proxy server code
- implement enable/disable of the forward proxy in the e2e test
The end result is shown as the following diagram:
```
A -- A's forward proxy (connections initiated from A will be forward from this proxy)
| ^ covers case (b)
|
--- A's reverse proxy (advertised to other peers where the connection should come in from)
^ covers case (a)
```
[Test]
make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1
make gofail-enable && make build && make gofail-disable && \
go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1
[Known issue]
I run into `context deadline exceeded` sometimes.
[References]
[1] Issue etcd-io#17737
[2] Supersedes PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] Superseded PR (V2) etcd-io#17891
Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 6, 2024
Based on Fu Wei's idea discussed in the [issue](etcd-io#17737), we employ the blocking on L7 but without using external tools. A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. We introduce an "HTTP proxy" for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's HTTP proxy ----- B's (currently existing) proxy - B ^ newly introduced ^ in the original codebase ``` By adding this HTTP proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. The main subtasks are - set up an environment variable `FORWARD_PROXY` - implement HTTP proxy by extending the existing proxy server code - implement enable/disable of the HTTP proxy in the e2e test The result is that for every peer, we will have the arch like this ``` A -- A's HTTP proxy (connections initiated from A will be forwarded from this proxy) | ^ covers case (b) | --- A's (currently existing) proxy (advertised to other peers where the connection should come in from) ^ covers case (a) ``` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - I run into `context deadline exceeded` sometimes ``` etcd_mix_versions_test.go:175: Error Trace: /Users/henrybear327/go/src/etcd/tests/e2e/etcd_mix_versions_test.go:175 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:75 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:31 Error: Received unexpected error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE Test: TestBlackholeByMockingPartitionLeader Messages: failed to put "key-0", error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE ``` [References] [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 6, 2024
Thanks to Fu Wei for the input regarding etcd-io#17938. [Problem] A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Main idea] We introduce 1 shared "HTTP proxy" for all peers. All peers will be proxying all the connections through it. The modified architecture will look something like this: ``` A -- shared HTTP proxy ----- B ^ newly introduced ``` By adding this HTTP proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `FORWARD_PROXY`, because go will not parse HTTP_PROXY and HTTPS_PROXY that is using localhost or 127.0.0.1, regardless if the port is present or not - implement the shared HTTP proxy by extending the existing proxy server code - remove existing proxy setup (the per-peer proxy) - implement enable/disable of the HTTP proxy in the e2e test [Testing] - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` [References] [1] Tracking issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 [4] PR (V3) etcd-io#17938 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 6, 2024
Thanks to Fu Wei for the input regarding etcd-io#17938. [Problem] A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Main idea] We introduce 1 shared HTTP proxy for all peers. All peers will be proxying all the connections through it. The modified architecture will look something like this: ``` A -- shared HTTP proxy ----- B ^ newly introduced ``` By adding this HTTP proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `FORWARD_PROXY`, because go will not parse HTTP_PROXY and HTTPS_PROXY that is using localhost or 127.0.0.1, regardless if the port is present or not - implement the shared HTTP proxy by extending the existing proxy server code (we need to be able to identify the source sender) - remove existing proxy setup (the per-peer proxy) - implement enable/disable of the HTTP proxy in the e2e test [Testing] - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` [References] [1] Tracking issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 [4] PR (V3) etcd-io#17938 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 11, 2024
Based on Fu Wei's idea discussed in the [issue](etcd-io#17737), we employ the blocking on L7 but without using external tools. A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. We introduce an "HTTP proxy" for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's HTTP proxy ----- B's (currently existing) proxy - B ^ newly introduced ^ in the original codebase ``` By adding this HTTP proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. The main subtasks are - set up an environment variable `FORWARD_PROXY` - implement HTTP proxy by extending the existing proxy server code - implement enable/disable of the HTTP proxy in the e2e test The result is that for every peer, we will have the arch like this ``` A -- A's HTTP proxy (connections initiated from A will be forwarded from this proxy) | ^ covers case (b) | --- A's (currently existing) proxy (advertised to other peers where the connection should come in from) ^ covers case (a) ``` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - I run into `context deadline exceeded` sometimes ``` etcd_mix_versions_test.go:175: Error Trace: /Users/henrybear327/go/src/etcd/tests/e2e/etcd_mix_versions_test.go:175 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:75 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:31 Error: Received unexpected error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE Test: TestBlackholeByMockingPartitionLeader Messages: failed to put "key-0", error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE ``` [References] [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 11, 2024
Based on Fu Wei's idea discussed in the [issue](etcd-io#17737), we employ the blocking on L7 but without using external tools. A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. We introduce an "HTTP proxy" for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's HTTP proxy ----- B's (currently existing) proxy - B ^ newly introduced ^ in the original codebase ``` By adding this HTTP proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. The main subtasks are - set up an environment variable `FORWARD_PROXY` - implement HTTP proxy by extending the existing proxy server code - implement enable/disable of the HTTP proxy in the e2e test The result is that for every peer, we will have the arch like this ``` A -- A's HTTP proxy (connections initiated from A will be forwarded from this proxy) | ^ covers case (b) | --- A's (currently existing) proxy (advertised to other peers where the connection should come in from) ^ covers case (a) ``` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - I run into `context deadline exceeded` sometimes ``` etcd_mix_versions_test.go:175: Error Trace: /Users/henrybear327/go/src/etcd/tests/e2e/etcd_mix_versions_test.go:175 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:75 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:31 Error: Received unexpected error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE Test: TestBlackholeByMockingPartitionLeader Messages: failed to put "key-0", error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE ``` [References] [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 11, 2024
Thanks to Fu Wei for the input regarding etcd-io#17938. [Problem] A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Main idea] We introduce 1 shared HTTP proxy for all peers. All peers will be proxying all the connections through it. The modified architecture will look something like this: ``` A -- shared HTTP proxy ----- B ^ newly introduced ``` By adding this HTTP proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `FORWARD_PROXY`, because go will not parse HTTP_PROXY and HTTPS_PROXY that is using localhost or 127.0.0.1, regardless if the port is present or not - implement the shared HTTP proxy by extending the existing proxy server code (we need to be able to identify the source sender) - remove existing proxy setup (the per-peer proxy) - implement enable/disable of the HTTP proxy in the e2e test [Testing] - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` [References] [1] Tracking issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 [4] PR (V3) etcd-io#17938 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 11, 2024
Based on Fu Wei's idea discussed in the [issue](etcd-io#17737), we employ the blocking on L7 but without using external tools. A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. We introduce an "HTTP proxy" for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's HTTP proxy ----- B's (currently existing) proxy - B ^ newly introduced ^ in the original codebase ``` By adding this HTTP proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. The main subtasks are - set up an environment variable `FORWARD_PROXY` - implement HTTP proxy by extending the existing proxy server code - implement enable/disable of the HTTP proxy in the e2e test The result is that for every peer, we will have the arch like this ``` A -- A's HTTP proxy (connections initiated from A will be forwarded from this proxy) | ^ covers case (b) | --- A's (currently existing) proxy (advertised to other peers where the connection should come in from) ^ covers case (a) ``` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - I run into `context deadline exceeded` sometimes ``` etcd_mix_versions_test.go:175: Error Trace: /Users/henrybear327/go/src/etcd/tests/e2e/etcd_mix_versions_test.go:175 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:75 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:31 Error: Received unexpected error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE Test: TestBlackholeByMockingPartitionLeader Messages: failed to put "key-0", error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE ``` [References] [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 12, 2024
Based on Fu Wei's idea discussed in the [issue](etcd-io#17737), we employ the blocking on L7 but without using external tools. [Background] A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Proposed solution] We introduce an forward proxy for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's forward proxy ----- B's reverse proxy - B ^ newly introduced ^ in the original codebase (renamed) ``` By adding this forward proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `E2E_TEST_FORWARD_PROXY_IP` - implement forward proxy by extending the existing proxy server code - implement enable/disable of the forward proxy in the e2e test The result is that for every peer, we will have the arch like this ``` A -- A's forward proxy (connections initiated from A will be forwarded from this proxy) | ^ covers case (b) | --- A's (currently existing) reverse proxy (advertised to other peers where the connection should come in from) ^ covers case (a) ``` [Testing] - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` [Issues] - I run into `context deadline exceeded` sometimes ``` etcd_mix_versions_test.go:175: Error Trace: /Users/henrybear327/go/src/etcd/tests/e2e/etcd_mix_versions_test.go:175 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:75 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:31 Error: Received unexpected error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE Test: TestBlackholeByMockingPartitionLeader Messages: failed to put "key-0", error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE ``` [References] [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 15, 2024
Based on Fu Wei's idea discussed in the [issue](etcd-io#17737), we employ the blocking on L7 but without using external tools. [Background] A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Proposed solution] We introduce an forward proxy for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's forward proxy ----- B's reverse proxy - B ^ newly introduced ^ in the original codebase (renamed) ``` By adding this forward proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `E2E_TEST_FORWARD_PROXY_IP` - implement forward proxy by extending the existing proxy server code - implement enable/disable of the forward proxy in the e2e test The result is that for every peer, we will have the arch like this ``` A -- A's forward proxy (connections initiated from A will be forwarded from this proxy) | ^ covers case (b) | --- A's (currently existing) reverse proxy (advertised to other peers where the connection should come in from) ^ covers case (a) ``` [Testing] - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` [Issues] - I run into `context deadline exceeded` sometimes ``` etcd_mix_versions_test.go:175: Error Trace: /Users/henrybear327/go/src/etcd/tests/e2e/etcd_mix_versions_test.go:175 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:75 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:31 Error: Received unexpected error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE Test: TestBlackholeByMockingPartitionLeader Messages: failed to put "key-0", error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE ``` [References] [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 16, 2024
Based on Fu Wei's idea discussed in the [issue](etcd-io#17737), we employ the blocking on L7 but without using external tools. [Background] A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Proposed solution] We introduce an forward proxy for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's forward proxy ----- B's reverse proxy - B ^ newly introduced ^ in the original codebase (renamed) ``` By adding this forward proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. [Testing] - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` [Issues] - I run into `context deadline exceeded` sometimes ``` etcd_mix_versions_test.go:175: Error Trace: /Users/henrybear327/go/src/etcd/tests/e2e/etcd_mix_versions_test.go:175 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:75 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:31 Error: Received unexpected error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE Test: TestBlackholeByMockingPartitionLeader Messages: failed to put "key-0", error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE ``` [References] [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com> It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `E2E_TEST_FORWARD_PROXY_IP` - implement forward proxy by extending the existing proxy server code - implement enable/disable of the forward proxy in the e2e test The result is that for every peer, we will have the arch like this ``` A -- A's forward proxy (connections initiated from A will be forwarded from this proxy) | ^ covers case (b) | --- A's (currently existing) reverse proxy (advertised to other peers where the connection should come in from) ^ covers case (a) ```
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 18, 2024
Based on Fu Wei's idea discussed in the [issue](etcd-io#17737), we employ the blocking on L7 but without using external tools. [Background] A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Proposed solution] We introduce an forward proxy for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's forward proxy ----- B's reverse proxy - B ^ newly introduced ^ in the original codebase (renamed) ``` By adding this forward proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. [Testing] - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` [Issues] - I run into `context deadline exceeded` sometimes ``` etcd_mix_versions_test.go:175: Error Trace: /Users/henrybear327/go/src/etcd/tests/e2e/etcd_mix_versions_test.go:175 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:75 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:31 Error: Received unexpected error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE Test: TestBlackholeByMockingPartitionLeader Messages: failed to put "key-0", error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE ``` [References] [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com> It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `E2E_TEST_FORWARD_PROXY_IP` - implement forward proxy by extending the existing proxy server code - implement enable/disable of the forward proxy in the e2e test The result is that for every peer, we will have the arch like this ``` A -- A's forward proxy (connections initiated from A will be forwarded from this proxy) | ^ covers case (b) | --- A's (currently existing) reverse proxy (advertised to other peers where the connection should come in from) ^ covers case (a) ```
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 21, 2024
Based on Fu Wei's idea discussed in the [issue](etcd-io#17737), we employ the blocking on L7 but without using external tools. [Background] A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Proposed solution] We introduce an forward proxy for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's forward proxy ----- B's reverse proxy - B ^ newly introduced ^ in the original codebase (renamed) ``` By adding this forward proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. [Testing] - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` [Issues] - I run into `context deadline exceeded` sometimes ``` etcd_mix_versions_test.go:175: Error Trace: /Users/henrybear327/go/src/etcd/tests/e2e/etcd_mix_versions_test.go:175 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:75 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:31 Error: Received unexpected error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE Test: TestBlackholeByMockingPartitionLeader Messages: failed to put "key-0", error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE ``` [References] [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com> It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `E2E_TEST_FORWARD_PROXY_IP` - implement forward proxy by extending the existing proxy server code - implement enable/disable of the forward proxy in the e2e test The result is that for every peer, we will have the arch like this ``` A -- A's forward proxy (connections initiated from A will be forwarded from this proxy) | ^ covers case (b) | --- A's (currently existing) reverse proxy (advertised to other peers where the connection should come in from) ^ covers case (a) ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Based on the idea from @fuweid, this PR attempts to employ blocking on L7 but without using external tools.
Main ideas
The main idea is
X-PeerURLsfield from the header, as we know all traffic to peer will contain this header field.Based on the main ideas, we introduce an SSL termination proxy so we can obtain the
X-PeerURLsfield from the header.PoC Implementation Issues
There are 2 known issues with this approach
For 1), this way of blocking (by utilizing
X-PeerURLsfrom the header) will miss certain types of traffic to certain endpoints, as the traffic to some endpoints doesn't have theX-PeerURLsfield. Currently, the known ones are:/members,/version, and/raft/probing.As you can see from the log, its header doesn't contain the
X-PeerURLsfield, but only the following fields:map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]] /membersmap[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]] /versionmap[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]] /raft/probingFor 2), in order to read out
X-PeerURLsfrom the header, we need to terminate the SSL connection, as we can't drop cleartext traffic (ref [1]). Thus, a new optione2e.WithSSLTerminationProxy(true)is introduced, which will change the network flow intoKnown improvements required before turning RFC into PR
The prototype needs to be further improved for code review after fixing the following issues:
blackholeTXorblackholeRXwill black both TX and RX traffic instead of just the specified one.References:
[1] #15595
Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.