HTTP3 Client Certificate 421

[YC]

With the following configuration (hostnames replaced), and accessing over HTTP3, Caddy responds 421 for subdomain1.
I think this is a bug because subdomain1 doesn't have client_auth enabled.

subdomain1.tld {
        respond * 200 {
                body "Hi"
        }
}
subdomain2.tld {
        tls {
                client_auth {
                        mode require_and_verify
                        trusted_ca_cert_file /etc/caddy/ca.crt
                }
        }
        respond * 200 {
                body "Hi2"
        }
}

Without the tls directive accessing over h3 or with the tls directive accessing over h2, both subdomains can be accessed without issue.

With strict_sni_host insecure_off and the tls directive in place, subdomain1 can be accessed, and subdomain2 gets ERR_QUIC_PROTOCOL_ERROR. I suspect the ERR_QUIC_PROTOCOL_ERROR is probably due to my configuration or method of testing rather than Caddy though (chromium --origin-to-force-quic-on=subdomain1.tld:443,subdomain2.tld:443), given that the browser doesn't bring up the 'Select a certificate` prompt.

[Saklad5]

I'm noticing 421 responses over HTTP/3, without a client certificate, when using strict_sni_host. Is it possible that option is completely broken for HTTP/3?

[Saklad5]

With the following configuration (hostnames replaced)

It is recommended that you avoid redacting public information like hostnames. And if you really want to, you should be using the .example TLD set aside for placeholders.

[mholt]

Thanks for opening an issue! We'll look into this.

It's not immediately clear to me what is going on, so I'll need your help to understand it better.

Ideally, we need to be able to reproduce the bug in the most minimal way possible. This allows us to write regression tests to verify the fix is working. If we can't reproduce it, then you'll have to test our changes for us until it's fixed -- and then we can't add test cases, either.

I've attached a template below that will help make this easier and faster! This will require some effort on your part -- please understand that we will be dedicating time to fix the bug you are reporting if you can just help us understand it and reproduce it easily.

This template will ask for some information you've already provided; that's OK, just fill it out the best you can. 👍 I've also included some helpful tips below the template. Feel free to let me know if you have any questions!

Thank you again for your report, we look forward to resolving it!

Template

## 1. Environment

### 1a. Operating system and version

```
paste here
```


### 1b. Caddy version (run `caddy version` or paste commit SHA)

```
paste here
```


### 1c. Go version (if building Caddy from source; run `go version`)

```
paste here
```


## 2. Description

### 2a. What happens (briefly explain what is wrong)




### 2b. Why it's a bug (if it's not obvious)




### 2c. Log output

```
paste terminal output or logs here
```



### 2d. Workaround(s)




### 2e. Relevant links




## 3. Tutorial (minimal steps to reproduce the bug)

Instructions -- please heed otherwise we cannot help you (help us help you!)

1. Environment: Please fill out your OS and Caddy versions, even if you don't think they are relevant. (They are always relevant.) If you built Caddy from source, provide the commit SHA and specify your exact Go version.

2. Description: Describe at a high level what the bug is. What happens? Why is it a bug? Not all bugs are obvious, so convince readers that it's actually a bug.

   • 2c) Log output: Paste terminal output and/or complete logs in a code block. DO NOT REDACT INFORMATION except for credentials.
   • 2d) Workaround: What are you doing to work around the problem in the meantime? This can help others who encounter the same problem, until we implement a fix.
   • 2e) Relevant links: Please link to any related issues, pull requests, docs, and/or discussion. This can add crucial context to your report.

3. Tutorial: What are the minimum required specific steps someone needs to take in order to experience the same bug? Your goal here is to make sure that anyone else can have the same experience with the bug as you do. You are writing a tutorial, so make sure to carry it out yourself before posting it. Please:

   • Start with an empty config. Add only the lines/parameters that are absolutely required to reproduce the bug.
   • Do not run Caddy inside containers.
   • Run Caddy manually in your terminal; do not use systemd or other init systems.
   • If making HTTP requests, avoid web browsers. Use a simpler HTTP client instead, like curl.
   • Do not redact any information from your config (except credentials). Domain names are public knowledge and often necessary for quick resolution of an issue!
   • Note that ignoring this advice may result in delays, or even in your issue being closed. 😞 Only actionable issues are kept open, and if there is not enough information or clarity to reproduce the bug, then the report is not actionable.

Example of a tutorial:

Create a config file:

{ ... }

Open terminal and run Caddy:

$ caddy ...

Make an HTTP request:

$ curl ...

Notice that the result is ___ but it should be ___.

[Saklad5]

@mholt Have you considered making that template into an issue form, rather than manually replying with it each time?

I could probably set it up in a pull request, if you wish.

[mholt]

@Saklad5 It's a canned response. We only use it when it's helpful. Some issues don't need the bug report template. And we know from experience with our forum that people don't follow templates or will fill out bogus information to bypass validations. We basically only have our maintainers triage issues, rather than having users do that themselves. We have a much higher success rate on GitHub in terms of compliance to templates.

Basically, I just need to know how to reproduce this with simple instructions. For instance, is chromium --origin-to-force-quic-on=... really necessary to reproduce the issue? If not, maybe it is an issue with Chromium? Right now there are too many factors, too many redacted pieces, and not enough logs.

[Saklad5]

Would you like me to fill out the template myself? I may have experienced the same issue using Safari. I've disabled strict SNI checking for the moment in response, but I can re-enable it upon request so you can reproduce it.

[mholt]

That might be helpful, thanks! I would also be interested in what is in your logs.

[francislavoie]

This is related to #4294

I think the problem is we don't actually know the SNI early in h3 because the handshake is async, is my understanding.

[Saklad5]

1. Environment

1a. Operating system and version

Ubuntu 18.04.6 LTS

1b. Caddy version (run caddy version or paste commit SHA)

v2.6.1 h1:EDqo59TyYWhXQnfde93Mmv4FJfYe00dO60zMiEt+pzo=

2. Description

2a. What happens (briefly explain what is wrong)

HTTP/3 requests are responded to with a 421 status code, as if they had an invalid SNI.

2b. Why it's a bug (if it's not obvious)

The SNI seems correct, and this only occurs over HTTP/3. I can't imagine a legitimate reason that this wouldn't occur over HTTP/2, so I believe the bug lies with Caddy.

2c. Log output

{"level":"debug","ts":1664046013.5690753,"logger":"events","msg":"event","name":"tls_get_certificate","id":"c5fbdf2f-0814-4e85-be1c-0323fa3e3801","origin":"tls","data":{"client_hello":{"CipherSuites":[60138,4865,4866,4867],"ServerName":"saklad5.com","SupportedCurves":[14906,29,23,24,25],"SupportedPoints":null,"SignatureSchemes":[1027,2052,1025,1283,515,2053,2053,1281,2054,1537,513],"SupportedProtos":["h3"],"SupportedVersions":[23130,772],"Conn":{}}}}
{"level":"debug","ts":1664046013.569236,"logger":"tls.handshake","msg":"choosing certificate","identifier":"saklad5.com","num_choices":1}
{"level":"debug","ts":1664046013.5692725,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"saklad5.com","subjects":["saklad5.com"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"9118831e6229e304b9326866a978c787d02a3c0b047ff030fe2368d1a5e33149"}
{"level":"debug","ts":1664046013.5692906,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"2001:db8:ffff:ffff:ffff:ffff:ffff:ffff","remote_port":"60313","subjects":["saklad5.com"],"managed":true,"expiration":1670006168,"hash":"9118831e6229e304b9326866a978c787d02a3c0b047ff030fe2368d1a5e33149"}
{"level":"debug","ts":1664046013.6953883,"logger":"events","msg":"event","name":"tls_get_certificate","id":"08d55306-5f12-471f-8d92-694a22c89437","origin":"tls","data":{"client_hello":{"CipherSuites":[47802,4865,4866,4867],"ServerName":"saklad5.com","SupportedCurves":[6682,29,23,24,25],"SupportedPoints":null,"SignatureSchemes":[1027,2052,1025,1283,515,2053,2053,1281,2054,1537,513],"SupportedProtos":["h3"],"SupportedVersions":[51914,772],"Conn":{}}}}
{"level":"debug","ts":1664046013.6954954,"logger":"tls.handshake","msg":"choosing certificate","identifier":"saklad5.com","num_choices":1}
{"level":"debug","ts":1664046013.6955202,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"saklad5.com","subjects":["saklad5.com"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"9118831e6229e304b9326866a978c787d02a3c0b047ff030fe2368d1a5e33149"}
{"level":"debug","ts":1664046013.695557,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"192.0.2.0","remote_port":"57879","subjects":["saklad5.com"],"managed":true,"expiration":1670006168,"hash":"9118831e6229e304b9326866a978c787d02a3c0b047ff030fe2368d1a5e33149"}
{"level":"debug","ts":1664046013.6989307,"logger":"http.log.error","msg":"strict host matching: TLS ServerName () and HTTP Host (saklad5.com) values differ","request":{"remote_ip":"2001:db8:ffff:ffff:ffff:ffff:ffff:ffff","remote_port":"60313","proto":"HTTP/3.0","method":"GET","host":"saklad5.com","uri":"/.well-known/security.txt","headers":{"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15"],"Accept-Language":["en-US,en;q=0.9"],"Accept-Encoding":["gzip, deflate, br"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],"Priority":["u=0, i"],"Upgrade-Insecure-Requests":["1"]},"tls":{"resumed":false,"version":0,"cipher_suite":0,"proto":"","server_name":""}},"duration":0.001085679,"status":421,"err_id":"88yvk1g40","err_trace":"caddyhttp.(*Server).enforcementHandler (server.go:367)"}

I have the full log, but this is the part that seems specific to the request. If you think there may be more to see, let me know.

I've redacted the remote IP addresses, since my current location is irrelevant.

2d. Workaround(s)

Disabling HTTP/3 (or simply opting not to use it when connecting) avoids the issue, as does removing strict_sni_host.

3. Tutorial (minimal steps to reproduce the bug)

1. Enable HTTP/3 under "Develop → Experimental Features" in Safari 16.0 on macOS.
2. Visit https://saklad5.com/.well-known/security.txt.

Disabling HTTP/3 will cause the error to vanish.

---

With other domains excised (it continues after the first site block), the following is my Caddyfile:

{
        debug

        log {
                output file /var/log/caddy/access.log {
                        roll_disabled
                        roll_uncompressed
                }
        }

        auto_https disable_redirects

        acme_ca https://acme-v02.api.letsencrypt.org/directory
        email admin@saklad5.com

        admin unix//run/caddy/caddy-admin.sock

        servers {
                protocols h1 h2 h2c h3
                strict_sni_host
        }
}

(global) {
        @https {
                protocol https
        }
        @unsupported {
                method POST PUT DELETE CONNECT OPTIONS TRACE
        }

        bind 159.223.125.173

        header @https ?Strict-Transport-Security "max-age=63072000; includeSubDomains"
        header ?Content-Security-Policy "base-uri 'none' default-src 'none'"
        header ?Cache-Control "max-age=600; stale-while-revalidate=85800"
        encode zstd br gzip
        error @unsupported 501
}

(files) {
        file_server {
                root /var/www/{args.0}
        }
}

(onion) {
        header @https Alt-Svc "h3=\":443\"; ma=2592000; persist=1, h2=\"{args.1}:443\"; ma=7776000; persist=1"
        header @https Onion-Location "http://{args.0}.{args.1}{http.request.uri.path}"
}

https://saklad5.com http://saklad5.com.{env.ONION__} {
        import global
        import files @
        import onion saklad5.com {env.ONION__}

        bind ::1 2604:a880:400:d0::2204:1

        header @https Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
}

---

Safari relies on the draft HTTPS DNS record to use HTTP/3, in case that's relevant. If you don't feel like manually decoding the wire format, I can give you those records in presentation format.

Given that this is an experimental feature, it is not impossible that the bug is actually Safari's fault, and the Chrome issue is unrelated.

[Saklad5]

This is related to #4294

I think the problem is we don't actually know the SNI early in h3 because the handshake is async, is my understanding.

I didn't notice that issue, probably because I was searching for 421 errors. I agree, that's almost certainly the same problem. According to the log I just posted, the TLS ServerName Caddy is evaluating is empty. This might be one of those classic Go race conditions.

[Saklad5]

Let me know if I can turn secure_sni_host off. It may be an experimental feature right now, but I still don't want to leave this broken on my server for too long.

[francislavoie]

I didn't notice that issue, probably because I was searching for 421 errors.

That's because we actually merged a pull request to change the strict_sni_host error from status 403 to 421: #4023

[YC]

1. Environment

1a. Operating system and version

Ubuntu 22.04.1 LTS

1b. Caddy version (run caddy version or paste commit SHA)

v2.6.1 h1:EDqo59TyYWhXQnfde93Mmv4FJfYe00dO60zMiEt+pzo=

2. Description

2a. What happens (briefly explain what is wrong)

421 for hostname without client authentication enabled.

2b. Why it's a bug (if it's not obvious)

The client should able to access the hostname without client authentication enabled over h3 without issue.

2c. Log output

{"level":"debug","ts":1664086030.5471375,"logger":"tls.handshake","msg":"choosing certificate","identifier":"subdomain1.steventang.net","num_choices":1}
{"level":"debug","ts":1664086030.547153,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"subdomain1.steventang.net","subjects":["subdomain1.steventang.net"],"managed":true,"issuer_key":"acme.zerossl.com-v2-DV90","hash":"6e1f5d3713b1f47acbca9dfd3497b4a0c152e5f120723e0078c4da8d3c921c72"}
{"level":"debug","ts":1664086030.5471725,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"<redacted>","remote_port":"46343","subjects":["subdomain1.steventang.net"],"managed":true,"expiration":1671926400,"hash":"6e1f5d3713b1f47acbca9dfd3497b4a0c152e5f120723e0078c4da8d3c921c72"}
{"level":"debug","ts":1664086030.5798016,"logger":"http.log.error","msg":"strict host matching: TLS ServerName () and HTTP Host (subdomain1.steventang.net) values differ","request":{"remote_ip":"<redacted>","remote_port":"46343","proto":"HTTP/3.0","method":"GET","host":"subdomain1.steventang.net","uri":"/","headers":{"Accept":["*/*"],"User-Agent":["curl/7.76.1-DEV"]},"tls":{"resumed":false,"version":0,"cipher_suite":0,"proto":"","server_name":""}},"duration":0.00003672,"status":421,"err_id":"yhcnism9u","err_trace":"caddyhttp.(*Server).enforcementHandler (server.go:367)"}

$ docker run --rm ymuski/curl-http3 curl --http3 -vvv https://subdomain1.steventang.net
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 152.67.104.138:443...
* Connect socket 5 over QUIC to 152.67.104.138:443
* Sent QUIC client Initial, ALPN: h3-29,h3-28,h3-27
* Connected to subdomain1.steventang.net () port 443 (#0)
* h3 [:method: GET]
* h3 [:path: /]
* h3 [:scheme: https]
* h3 [:authority: subdomain1.steventang.net]
* h3 [user-agent: curl/7.76.1-DEV]
* h3 [accept: */*]
* Using HTTP/3 Stream ID: 0 (easy handle 0x555e52fb11c0)
> GET / HTTP/3
> Host: subdomain1.steventang.net
> user-agent: curl/7.76.1-DEV
> accept: */*
>
< HTTP/3 421
< server: Caddy
< alt-svc: h3=":443"; ma=2592000
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host subdomain1.steventang.net left intact

$ ./curl -vvv --http3 https://subdomain1.steventang.net --cacert ~/Desktop/ca/certs/ca.cert.pem --cert ~/Desktop/ca/intermediate/certs/client.cert.pem --key ~/Desktop/ca/intermediate/private/client.key.pem
*   Trying 152.67.104.138:443...
*  CAfile: /home/steve/Desktop/ca/certs/ca.cert.pem
*  CApath: none
* Connect socket 5 over QUIC to 152.67.104.138:443
* Sent QUIC client Initial, ALPN: h3,h3-29,h3-28,h3-27
*  subjectAltName: host "subdomain1.steventang.net" matched cert's "subdomain1.steventang.net"
* Verified certificate just fine
* Connected to subdomain1.steventang.net (152.67.104.138) port 443 (#0)
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: subdomain1.steventang.net]
* h2h3 [user-agent: curl/7.86.0-DEV]
* h2h3 [accept: */*]
* Using HTTP/3 Stream ID: 0 (easy handle 0x558ba7a50d80)
> GET / HTTP/3
> Host: subdomain1.steventang.net
> user-agent: curl/7.86.0-DEV
> accept: */*
>
< HTTP/3 421
< server: Caddy
< alt-svc: h3=":443"; ma=2592000

2d. Workaround(s)

• Don't use h3
• Remove client authentication on h3
• Add strict_sni_host insecure_off

2e. Relevant links

#4294
#5078 (comment) above which seems like a more minimal reproduction with simply strict_sni_host on

3. Tutorial (minimal steps to reproduce the bug)

Set up server with one hostname without client authentication and one hostname with client authentication.
Attempt to access hostname without client authentication over h3.

{
	email admin@steventang.net

	log {
		output file /var/log/caddy/caddy.log
		level DEBUG
	}

	admin off
}


subdomain1.steventang.net {
	respond 200 {
		body "subdomain1"
	}
}
subdomain2.steventang.net {
        tls {
                client_auth {
                        mode require_and_verify
                        trusted_ca_cert_file /etc/caddy/ca.crt
                }
        }
	respond 200 {
		body "subdomain2"
	}
}

[WeidiDeng]

strict_sni_host is incompatible with http3 because of stdlib tls state handling, that will depend on stdlib fix.

caddy currently will enable strict_sni_host whenever tls_auth is enabled. Maybe we should track which hostname to check for sni matching in this case @mholt.

[mholt]

@WeidiDeng

Maybe we should track which hostname to check for sni matching in this case

That might be possible. Trying to think if that opens up any vulnerabilities.

[Saklad5]

Would it be possible (or even standards-compliant) to enable strict SNI matching on everything except HTTP/3? Just as an interim fix?

I'm starting to think enabling HTTP/3 by default was premature.

[mholt]

That would leave your site vulnerable over HTTP/3, so probably not a good idea.

Client auth is used by very, very few servers, and even fewer clients over HTTP/3. Rather than wait around for dependencies to support everything, I'd rather get what we can working in the meantime.

[Saklad5]

That would leave your site vulnerable over HTTP/3, so probably not a good idea.

Agreed, but that's still better than leaving it off entirely.

---

Once the ECH spec is finalized, by the way, I hope Caddy will implement support. That makes domain fronting obsolete, which hopefully means you can make strict SNI matching the default.

Personally I think it should be the default now, given the SNI spec requires that behavior anyway, but I'm guessing it caused enough problems for enough people to justify the current approach.

[mholt]

Agreed, but that's still better than leaving it off entirely.

What? 🤔

No, HTTP/3 is definitely very, very optional.

Once the ECH spec is finalized, by the way, I hope Caddy will implement support.

I hope the Go standard library will support it! (That's also what is holding back this issue.)

[PaddyPat]

Any news about this issue?
It’s the main reason for me to stay on old caddy version.
Br

[mholt]

I don't think so. @PaddyPat You can simply disable HTTP/3 if it's holding you back. (Use of client certs like this over H3 is still a niche use case...)

[PaddyPat]

I don't think so. @PaddyPat You can simply disable HTTP/3 if it's holding you back. (Use of client certs like this over H3 is still a niche use case...)

You mean:

servers {
protocols h1 h2 h2c
strict_sni_host
}

?

[francislavoie]

Yeah - you probably don't need to enable h2c though, just specify h1 and h2, to disable h3. Enabling h2c only makes sense for port 80, and only if you have a specific need for it. Most don't, because TLS is trivial with Caddy and h2c is only needed when the client can only do h2 but not h1, e.g. with gRPC.

[PaddyPat]

disabled h3 but the issue till exist:
tls -> mode require_and_verify (as written above - 3. Tutorial (minimal steps to reproduce the bug))

ERR_BAD_SSL_CLIENT_AUTH_CERT

I use Authelia as middleware.
Access portal.domain.com requires tls cert and than Authelia login via auth.domain.com

Why? I have one ip allowed for multiple users but only users with local installed cert should reach auth.Domain.com.

other subdomains / services doesn’t require tls so other users can use them

[mholt]

What is the output of curl -v? Or better yet, curl --trace-ascii.

[PaddyPat]

here we go...
I've disabled for this test the authelia middleware for this domain so it can't be authelia

with caddy 2.6 (latest)

curl -v -k https://stats.domain.de:443 --cert cert_domain_de.pem --key cert_domain_de.key
*   Trying 181.xx.xx.xx:443...
* Connected to stats.domain.de (181.xx.xx.xx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
Enter PEM pass phrase:
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, CERT verify (15):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_CHACHA20_POLY1305_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=stats.domain.de
*  start date: Dec  7 20:16:22 2022 GMT
*  expire date: Mar  7 20:16:21 2023 GMT
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* OpenSSL SSL_write: Connection reset by peer, errno 104
* Failed sending HTTP2 data
* Connection #0 to host stats.domain.de left intact
curl: (16) OpenSSL SSL_write: Connection reset by peer, errno 104

with caddy 2.5.2 (without h1 h2 strict_sni)

curl -v -k https://stats.domain.de:443 --cert cert_domain_de.pem --key cert_domain_de.key
*   Trying 181.xx.xx.xx:443...
* Connected to stats.domain.de (181.xx.xx.xx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
Enter PEM pass phrase:
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, CERT verify (15):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_CHACHA20_POLY1305_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=stats.domain.de
*  start date: Dec  7 20:16:22 2022 GMT
*  expire date: Mar  7 20:16:21 2023 GMT
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x56396a08c4d0)
> GET / HTTP/2
> Host: stats.domain.de
> user-agent: curl/7.74.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 403
< server: Caddy
< content-length: 0
< date: Fri, 13 Jan 2023 19:03:39 GMT
<
* Connection #0 to host stats.domain.de left intact

[mholt]

@PaddyPat Thanks! And are you getting the same server logs as posted above by @YC? Can you post your server logs when you make that request? (Preferably with debug-level logging enabled)

[PaddyPat]

there is no log entry created,..
I select the cert and than its finished:
ERR_BAD_SSL_CLIENT_AUTH_CERT

log works, if I use the caddy 2.5.2 so it's not an issue from me by implementing log debug in caddy

[mholt]

No log entry, even with debug logging? That's surprising to me. Either the request is not hitting your server as you think, or we have a hole in our log coverage... but I would think that even the std lib should be outputting something to debug logs...

If you don't have logs, then it means your issue is different than the OP / what this thread is about, because this thread at least has logs. (We know the problem and the solution, but it has to be fixed in the Go std lib.)

[PaddyPat]

curl -v -k https://stats.domain.de:443 --cert cert_domain_de.pem --key cert_domain_de.key
*   Trying 181.xx.xx.xx:443...
* Connected to stats.domain.de (181.xx.xx.xx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
Enter PEM pass phrase:
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, CERT verify (15):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_CHACHA20_POLY1305_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=stats.domain.de
*  start date: Dec  7 20:16:22 2022 GMT
*  expire date: Mar  7 20:16:21 2023 GMT
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5651114284d0)
> GET / HTTP/2
> Host: stats.domain.de
> user-agent: curl/7.74.0
> accept: */*
>
* TLSv1.3 (IN), TLS alert, bad certificate (554):
* OpenSSL SSL_read: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate, errno 0
* Failed receiving HTTP2 data
* OpenSSL SSL_write: SSL_ERROR_ZERO_RETURN, errno 0
* Failed sending HTTP2 data
* Connection #0 to host stats.domain.de left intact
curl: (56) OpenSSL SSL_read: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate, errno 0

logfile:

{"level":"error","ts":1673638171.0985782,"logger":"http.log.access.log11","msg":"handled request","request":{"remote_ip":"181.xx.xx.xx","remote_port":"40238","proto":"HTTP/2.0","method":"GET","host":"stats.domain.de","uri":"/","headers":{"User-Agent":["curl/7.74.0"],"Accept":["/"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","server_name":"stats.domain.de"}},"user_id":"","duration":0.000097595,"size":0,"status":403,"resp_headers":{"Server":["Caddy"],"Content-Type":[]}}

[mholt]

@PaddyPat That looks like a different issue than this thread is about. Your connection is using HTTP/2 and is returning a 403, whereas this thread is about the TLS ServerName being empty in HTTP/3 connections: TLS ServerName () and HTTP Host (saklad5.com) values differ

Yours looks more like the client actually is providing the wrong certificate (or the client auth is misconfigured). Please open a new issue and, preferably, fill out the bug report template I posted above. Or ask your question in the forums so we can help you out. Thanks!

[WeidiDeng]

After upgrading quic-go, this problem is no more in latest caddy.