reverseproxy.go:490: suppressing panic for copyResponse error in test; #8927
Comments
nodje
changed the title
|
More of it the next day, ending in a I'm going to activate debug log.level, hopefully the pattern will reproduce and more clues will come out of it. |
|
For some reason these messages don't show in the tarefik.log, only through I assume docker logs output and the content of /mnt/data/traefik/log/traefik.log should be the same. The message |
mpl
added
kind/bug/possible
|
This issue is still present in the 2.8 branch of Traefik and appears to be related to the Go standard library implementation of the reverse proxy. It comes from here: https://cs.opensource.google/go/go/+/master:src/net/http/httputil/reverseproxy.go;l=495 This error is spammed to the stdout only because the reverse proxy is invoking Go's default log package to report the error. There are several ways I can imagine that this issue could be fixed:
Personally I think option 3 is preferable as it mimics the behaviour of the standard library and allows Traefik to react to a request streaming error and output a properly formatted log message. Looking through the code for httputil, setting a context value for http.ServerContextKey should not cause an issue, but that would need to be tested in case there is some upstream issue in a chained handler or something. |
|
good job @niallnsec Could you elaborate on what you said here:
AFAIUm, it's a downstreajm system responding a streaming error, and Traefik should know better on how to catch and log that rather than letting the underlying Go httputil log directly to stdout? Correct? |
|
Hello @nodje , @niallnsec , I have been able to reproduce an equivalent situation by making a backend dying right in the middle of it writing its response. However, from what I can see (I sprinkled some debugs both in traefik and in the stdlib), everything seems to behave as it should, i.e.:
So at the moment I'm puzzled as to why you're not seeing them same behaviour. I was running traefik directly from the binary, built on master at 703de53 WDYT? Edit: I can confirm that I observe the same behaviour as described above with traefik v2.8 @ 1c9a7b8 |
|
Hi @mpl I have been trying to build a test container I can run in order to try and get a stack trace to share, I made a small modification to the traefik main function below as a rough debug measure: type traceWriter struct {
}
func (t traceWriter) Write(p []byte) (n int, err error) {
debug.PrintStack()
return os.Stderr.Write(p)
}
var _ io.Writer = traceWriter{}
func main() {
stdlog.Default().SetOutput(traceWriter{})However, no matter what I do I cannot get a traefik container to build on my system. I suspect it has something to do with the fact that I am using buildx and am on an Apple M1 processor. I am deploying traefik into an x86 environment though. Would you be able to help with building a test container? If so I can deploy it and get some more info. |
|
Hi @niallnsec
To build a Traefik container for amd64, you should use the following command: |
|
Hi! I'm Træfiker 🤖 the bot in charge of tidying up the issues. I have to close this one because of its lack of activity 😞 Feel free to re-open it or join our Community Forum. |
|
I don't think the issue is solved. I'm not sure how to re-open the issue. |
|
Hey @nodje, I have reopened the issue for you. We have not been able to reproduce this bug in our environment and it looks like time to get more eyes on it. If any community member can help us find verify steps to reproduce, we would love the help. |
|
Hello, I've been receiving the same error with traefik v2.9.6 and I'm also running a media server (Plex). I'm going to try and create steps to reproduce the error. |
|
Getting same errors, (2.9.6 and 2.9.7). These are the errors I have in logs after the upgrade to 2.9.7: Just tried to reach my backend and got 502 Bad Gateway |
|
@tfny it's caused by quic-go and also happens in caddy Current assessment by @marten-seemann in caddyserver/caddy#5766 (comment) upstream issue: quic-go/quic-go#3737 |
|
Hey @bt90, Thank you for the information. We'll follow the PR that should fix the issue. Thus you'll be able to test it. |
|
Closed by #10137. |
Welcome!
What did you do?
After upgrading to latest 2.6.3 (could be unrelated, but it's the first time I notice this error), I found these errors in the logs:
17:46 Apr6 was when I restarted Traefik last time.
On that same day at night this happened.
It's a low usage instance, with few users only
Potentially, a movie was being streamed at the time (Jellyfin instance) but that's only a wild guess at what could have caused this. I don't even even understand how demanding streaming can be on a reverse proxy.
What did you see instead?
See
What version of Traefik are you using?
2.6.3
What is your environment & configuration?
If applicable, please paste the log output in DEBUG level
The text was updated successfully, but these errors were encountered: