stream xxx canceled by remote with error code 268

[bt90]

I often receive the following error in my caddy logs while doing speed tests with librespeed:

{
   "level":"error",
   "ts":1692452491.2814867,
   "logger":"http.handlers.reverse_proxy",
   "msg":"aborting with incomplete response",
   "upstream":"localhost:8085",
   "duration":0.005322624,
   "request":{
      "remote_ip":"192.168.178.20",
      "remote_port":"60436",
      "client_ip":"192.168.178.20",
      "proto":"HTTP/3.0",
      "method":"GET",
      "host":"speed.example.com",
      "uri":"/backend/garbage.php?r=0.6473048564721796&ckSize=100",
      "headers":{
         "Accept-Encoding":[
            "gzip, deflate, br"
         ],
         "Sec-Fetch-Mode":[
            "cors"
         ],
         "User-Agent":[
            "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/116.0"
         ],
         "X-Forwarded-For":[
            "192.168.178.20"
         ],
         "Referer":[
            "https://speed.example.como/speedtest_worker.js?r=0.8378941592896734"
         ],
         "Sec-Fetch-Dest":[
            "empty"
         ],
         "Sec-Fetch-Site":[
            "same-origin"
         ],
         "Cache-Control":[
            "no-cache"
         ],
         "Alt-Used":[
            "speed.example.com"
         ],
         "Accept-Language":[
            "de,en-US;q=0.7,en;q=0.3"
         ],
         "Dnt":[
            "1"
         ],
         "Pragma":[
            "no-cache"
         ],
         "Accept":[
            "*/*"
         ],
         "X-Forwarded-Proto":[
            "https"
         ],
         "X-Forwarded-Host":[
            "speed.example.com"
         ]
      },
      "tls":{
         "resumed":true,
         "version":772,
         "cipher_suite":4865,
         "proto":"h3",
         "server_name":"speed.example.com"
      }
   },
   "error":"writing: stream 80 canceled by remote with error code 268"
}

The mixture of HTTP/3 and librespeed canceling the requests seems to trigger this (bogus?) error.

[bt90]

You should be able to reproduce this with a reverse proxy to a public librespeed instance like https://librespeed.org/

[mholt]

The only other time I've heard of this error is here: https://caddy.community/t/error-stream-292-canceled-with-error-code-268/18847/6

Can you provide a minimally-reproducible config? Ideally, self-contained (i.e. use curl -v for a request, Caddy for the response). If it's only when proxying, try proxying to Caddy itself to isolate the bug to be within Caddy.

If it's not reproducible that way, it's likely a bug in some other part of the network or one of the other applications involved (client, backend, etc).

[bt90]

Example config:

example.com {
        reverse_proxy https://speed.piraten.tools {
                header_up Host {upstream_hostport}
        }
}

Open the page via Firefox and hit the Start button.

I suspect that the error is triggered by the client aborting the request while the response is still being transferred.

[bt90]

I wonder if this is an HTTP/3 error code. 268 in hex is 0x010C which is the assigned code for H3_REQUEST_CANCELLED.

https://www.iana.org/assignments/http3-parameters/http3-parameters.xhtml

[bt90]

The only other time I've heard of this error is here: https://caddy.community/t/error-stream-292-canceled-with-error-code-268/18847/6

Interesting. Traefik seems to also be affected: traefik/traefik#8927

As caddy and traefik are both using quic-go, the HTTP/3 theory sounds plausible.

@marten-seemann

[marten-seemann]

Pretty sure this is an HTTP/3 error. Request cancelations are nothing unexpected, but it seems like the error return value is less than ideal... this should have been immediately obvious.

Do you guys agree that this is not indicative of a bug, and the only action we should take is improve the error type / message?

The problem here is the split between the quic and the http3 package. The quic package doesn't know anything about the HTTP/3 layer, so it just sees an error 268. It would be the http3 layer's responsibility to catch this error and to replace (or at the very least wrap) it with something that looks more like an HTTP/3 error. RFC 9114 section 8 describes the error handling.

I'd appreciate feedback on the error API. What do you think of the following:

Any quic.ApplicationError and quic.StreamError would be replace with an http3.Error (better name, please?):

package http3

type Error struct {
     Remote bool // assuming there are no cases where we'd return a local error, we might be able to remove this field
     ErrorCode ErrorCode
     ErrorMessage string // always empty for stream resets, might be set if the connection is closed
}

The string representation of the error would then contain the name of the error (e.g. H3_REQUEST_CANCELLED for error 268).

[mholt]

Do you guys agree that this is not indicative of a bug, and the only action we should take is improve the error type / message?

Yep! Stream cancellations are usually not a problem -- in the past we've tried to sequestered them to debug-level logs when possible.

The string representation of the error would then contain the name of the error (e.g. H3_REQUEST_CANCELLED for error 268).

Would there be a way to use types to know this kind of error? For example errors.Is(...)?

Thanks for helping with this!

[marten-seemann]

The string representation of the error would then contain the name of the error (e.g. H3_REQUEST_CANCELLED for error 268).

Would there be a way to use types to know this kind of error? For example errors.Is(...)?

For errors.Is we'd have to introduce a new type for every error defined in the RFC: https://datatracker.ietf.org/doc/html/rfc9114#name-http-3-error-codes.

With the proposal from #5766 (comment), you'd do the error assertion like this:

var http3Err *http3.Error
if errors.As(err, &http3Err) {
    if http3Err.ErrorCode == http3.ErrCodeRequestCanceled {
          // request cancelled
    }
}

Any preferences?

[mholt]

Ah, that works for me then! I like the error code approach. (I guess that's the point of an error code.)

[marten-seemann]

Great! Implementation is here: quic-go/quic-go#4039. Planned for the v0.39 release.

[mholt]

That was quic, thank you!

[bt90]

@mholt v0.39 is available with the fix

[mholt]

I will update it shortly :) Thanks! (Been busy on a new Caddy website/docs.)

[bt90]

Is this resolved with the update of quic-go in 88b4fbf or does this need some additional adjustments in the logging logic?

[francislavoie]

We haven't made any changes to the error handling as per #5766 (comment) yet. PRs welcome, if you'd like to take a shot at it