egressgateway logs HTTP/2" 0 DPE, proxy ws server not work

[wenhx8]

(NOTE: This is used to report product bugs:
To report a security vulnerability, please visit https://istio.io/about/security-vulnerabilities
To ask questions about how to use Istio, please visit https://discuss.istio.io)

Bug description

[ ] Docs
[ ] Installation
[X] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
[ ] Upgrade

Expected behavior

Steps to reproduce the bug

Version (include the output of istioctl version --remote and kubectl version --short and helm version --short if you used Helm)

How was Istio installed?

Environment where the bug was observed (cloud vendor, OS, etc)

Additionally, please consider running istioctl bug-report and attach the generated cluster-state tarball to this issue.
Refer cluster state archive for more details.

curl info below:
About to connect() to istio-egressgateway.istio-system port 80 (#0)

• Trying 11.254.197.138...
• Connected to istio-egressgateway.istio-system (11.254.197.138) port 80 (#0)

GET /kvm/apis/subresources.kubevirt.io/v1alpha3/namespaces/chencheng-706706594914/virtualmachineinstances/vmnode/vnc HTTP/1.1
User-Agent: curl/7.29.0
Host: istio-egressgateway.istio-system
Accept: /
Authorization: Bearer 006050a52f9e5a8bbde879548276baaa
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
Sec-WebSocket-Key: WAeLeoMzEyAYjgdz1rQQvQ==
Sec-WebSocket-Protocol: binary
Sec-WebSocket-Version: 13
Upgrade: websocket
Connection: Upgrade

< HTTP/1.1 503 Service Unavailable
< content-length: 95
< content-type: text/plain
< date: Tue, 23 Feb 2021 02:51:57 GMT
< server: envoy
< connection: close
<

• Closing connection 0
  upstream connect error or disconnect/reset before headers. reset reason: connection termination

[howardjohn]

can you turn in debug logs of egress gateway? it should show more info about what is rejected

https://github.com/istio/istio/wiki/Troubleshooting-Istio

also maybe try curl the backend directly and see the responses. sometimes we have invalid http headers etc get rejected

[wenhx8]

can you turn in debug logs of egress gateway? it should show more info about what is rejected

https://github.com/istio/istio/wiki/Troubleshooting-Istio

also maybe try curl the backend directly and see the responses. sometimes we have invalid http headers etc get rejected
debug log
2021-02-21T07:25:21.140646Z debug envoy http2 [C35] updating connection-level initial window size to 268435456
2021-02-21T07:25:21.140654Z trace envoy http2 [C35] dispatching 1795 bytes
2021-02-21T07:25:21.140659Z trace envoy http2 [C35] about to recv frame type=4, flags=0
2021-02-21T07:25:21.140662Z trace envoy http2 [C35] track inbound frame type=4 flags=0 length=30 padding_length=0
2021-02-21T07:25:21.140667Z trace envoy http2 [C35] recv frame type=4
2021-02-21T07:25:21.140669Z trace envoy http2 [C35] about to recv frame type=8, flags=0
2021-02-21T07:25:21.140671Z trace envoy http2 [C35] track inbound frame type=8 flags=0 length=4 padding_length=0
2021-02-21T07:25:21.140674Z trace envoy http2 [C35] recv frame type=8
2021-02-21T07:25:21.140676Z trace envoy http2 [C35] about to recv frame type=1, flags=4
2021-02-21T07:25:21.140680Z trace envoy http2 [C35] track inbound frame type=1 flags=4 length=1710 padding_length=0
2021-02-21T07:25:21.140687Z debug envoy http [C35] new stream
2021-02-21T07:25:21.140717Z debug envoy http2 [C35] invalid frame: Invalid HTTP header field was received on stream 1
2021-02-21T07:25:21.140832Z debug envoy http [C35] dispatch error: The user callback function failed
2021-02-21T07:25:21.140839Z debug envoy http [C35][S5003666897163341861] stream reset
2021-02-21T07:25:21.140841Z trace envoy main item added to deferred deletion list (size=1)
2021-02-21T07:25:21.140845Z debug envoy connection [C35] closing data_to_write=0 type=2
2021-02-21T07:25:21.140848Z debug envoy connection [C35] setting delayed close timer with timeout 1000 ms
2021-02-21T07:25:21.140856Z trace envoy main clearing deferred deletion list (size=1)

[howardjohn]

well the error is pretty clear "Invalid HTTP header field was received on " but unfortunately doesn't show what the header was.

can you directly curl the service with -v?

[istio-policy-bot]

🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2021-02-23. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.

Created by the issue and PR lifecycle manager.

[lixianwa]

@howardjohn I had the same issus，the invalid header name is :authority, the detailed log below：

2021-09-18T02:11:53.363178Z	debug	envoy http2	[C2750] invalid http2: Invalid HTTP header field was received: frame type: 1, stream: 1, name: [:authority], value: [#initially=[etcd:2379]]
2021-09-18T02:11:53.363183Z	debug	envoy http2	[C2750] invalid frame: Invalid HTTP header field was received on stream 1
2021-09-18T02:11:53.363189Z	debug	envoy http	[C2750] dispatch error: The user callback function failed

Could you tell me how to solve it ？Thank you！

[howardjohn]

Its an etcd bug: etcd-io/etcd#13192