How to know if Tailscale cert integration was attempted?

[dcarrion87]

Trying to work out how caddy interacts with tailscale HTTPS mechanism and it's a bit of a mystery...

I have tailscale running in a kubernetes pod container. Running a tailscale cert works fine and produces cert and key file. Socket is at /tmp/tailscaled.socket by default in the sidecar.

As a test I bring the caddy binary into the container and run this to see if it interacts with socket directly:

caddy reverse-proxy --from REDCATED.REDCATED.ts.net --to localhost:3000

The logs don't indicate that they try to get a certificate via tailscale integration:

2022/09/16 04:50:01.201 WARN    admin   admin endpoint disabled
2022/09/16 04:50:01.201 INFO    http    server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "proxy", "https_port": 443}
2022/09/16 04:50:01.201 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "proxy"}
2022/09/16 04:50:01.202 INFO    autosaved config (load with --resume flag)      {"file": "/root/.config/caddy/autosave.json"}
Caddy proxying https://REDCATED.REDCATED.ts.net -> localhost:3000
2022/09/16 04:50:01.202 INFO    tls     cleaning storage unit   {"description": "FileStorage:/root/.local/share/caddy"}
2022/09/16 04:50:01.202 INFO    tls     finished cleaning storage units
2022/09/16 04:50:01.202 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc00043df10"}

Trying to access from another machine in the tailnet:

➜  ~ openssl s_client -connect REDCATED.REDCATED.ts.net:443
CONNECTED(00000006)
4486270636:error:14004438:SSL routines:CONNECT_CR_SRVR_HELLO:tlsv1 alert internal error:/System/Volumes/Data/SWE/macOS/BuildRoots/37599d3d49/Library/Caches/com.apple.xbs/Sources/libressl/libressl-56.60.4/libressl-2.8/ssl/ssl_pkt.c:1200:SSL alert number 80
4486270636:error:140040E5:SSL routines:CONNECT_CR_SRVR_HELLO:ssl handshake failure:/System/Volumes/Data/SWE/macOS/BuildRoots/37599d3d49/Library/Caches/com.apple.xbs/Sources/libressl/libressl-56.60.4/libressl-2.8/ssl/ssl_pkt.c:585:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 0 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Start Time: 1663303835
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---

How does one get more verbose info on the tailscale cert integration mechanism to see what it's actually doing?

[mholt]

Enable debug mode.

I just realized there's not a flag for that with the reverse-proxy command but I'll add that now. In the meantime, use a Caddyfile:

{
    debug
}

yours.you.ts.net

reverse_proxy :3000

Most common causes for failures are Caddy not having permission to read the socket or not restarting the TS service.

(PS. Stay tuned for something really cool to make this easier, soon!)

[mholt]

(There ya go, now you can use the --debug flag with the reverse-proxy command: b6cec37)

[dcarrion87]

From log file when request comes in:

022/09/16 05:13:21.444 INFO    using adjacent Caddyfile
...
2022/09/16 05:13:28.968 DEBUG   tls.handshake   no certificate matching TLS ClientHello {"server_name": "", "remote": "127.0.0.1:36496", "identifier": "127.0.0.1", "cipher_suites": [49200, 49196, 49192, 49188, 49172, 49162, 159, 107, 57, 52393, 52392, 52394, 65413, 196, 136, 129, 157, 61, 53, 192, 132, 49199, 49195, 49191, 49187, 49171, 49161, 158, 103, 51, 190, 69, 156, 60, 47, 186, 65, 49169, 49159, 5, 4, 49170, 49160, 22, 10, 255], "cert_cache_fill": 0, "load_if_necessary": true, "obtain_if_necessary": true, "on_demand": false}
2022/09/16 05:13:28.968 DEBUG   http.stdlib     http: TLS handshake error from 127.0.0.1:36496: no certificate available for '127.0.0.1'
2022/09/16 05:14:47.692 DEBUG   tls.handshake   no matching certificates and no custom selection logic  {"identifier": "REDACTED.REDACTED.ts.net"}
2022/09/16 05:14:47.692 DEBUG   tls.handshake   no matching certificates and no custom selection logic  {"identifier": "*.REDACTED.ts.net"}
2022/09/16 05:14:47.692 DEBUG   tls.handshake   no matching certificates and no custom selection logic  {"identifier": "*.*.ts.net"}
2022/09/16 05:14:47.692 DEBUG   tls.handshake   no matching certificates and no custom selection logic  {"identifier": "*.*.*.net"}
2022/09/16 05:14:47.692 DEBUG   tls.handshake   no matching certificates and no custom selection logic  {"identifier": "*.*.*.*"}
2022/09/16 05:14:47.692 DEBUG   tls.handshake   all external certificate managers yielded no certificates and no errors {"sni": "REDACTED.REDACTED.ts.net"}
2022/09/16 05:14:47.692 DEBUG   tls.handshake   no certificate matching TLS ClientHello {"server_name": "REDACTED.REDACTED.ts.net", "remote": "127.0.0.1:38648", "identifier": "REDACTED.REDACTED.ts.net", "cipher_suites": [49200, 49196, 49192, 49188, 49172, 49162, 159, 107, 57, 52393, 52392, 52394, 65413, 196, 136, 129, 157, 61, 53, 192, 132, 49199, 49195, 49191, 49187, 49171, 49161, 158, 103, 51, 190, 69, 156, 60, 47, 186, 65, 49169, 49159, 5, 4, 49170, 49160, 22, 10, 255], "cert_cache_fill": 0, "load_if_necessary": true, "obtain_if_necessary": true, "on_demand": false}
2022/09/16 05:14:47.692 DEBUG   http.stdlib     http: TLS handshake error from 127.0.0.1:38648: no certificate available for 'REDACTED.REDACTED.ts.net'

I can't see in the logs where it attempts to comms with the socket that's there.

When running tailscale cert on same system:

# tailscale --socket=/tmp/tailscaled.sock cert REDACTED.REDACTED.ts.net
Wrote public cert to REDACTED.REDACTED.ts.net.crt
Wrote private key to REDACTED.REDACTED.ts.net.key

Caddy version:

caddy version
v2.5.2 h1:eCJdLyEyAGzuQTa5Mh3gETnYWDClo1LjtQm2q9RNZrs=

[dcarrion87]

Also, not sure if this makes a difference but it's running on Kubernetes in tailscale sidecar as root (as a test).

Not sure if it's expecting anything else as part of the discovery process to determine if tailscale is running (which it is) and where the socket is.

[francislavoie]

Are you mounting the tailscale socket into your container? Caddy uses the socket to communicate with tailscale.

[dcarrion87]

Are you mounting the tailscale socket into your container? Caddy uses the socket to communicate with tailscale.

It's all happening in the same container for debugging / testing purposes. I.e. I'm trying to get caddy to talk with the tailscale socket that is present in the same container. But I can't even see if it's even attempting to from the log output.

As per #5041 (comment) when I run the tailscale cert option in the same container the socket stuff is working as it's bringing back a cert.

This is probably lack of understanding how caddy "detects" tailscale socket but there's nothing that shows in the log output that makes it obvious.

[francislavoie]

Did you see the notes in https://caddyserver.com/docs/automatic-https#activation about it? There might be some key insights in the links there that might help.

[dcarrion87]

Did you see the notes in https://caddyserver.com/docs/automatic-https#activation about it? There might be some key insights in the links there that might help.

The only note is:

Domains ending in .ts.net will not be managed by Caddy. Instead, Caddy will automatically attempt to get these certificates at handshake-time from the locally-running Tailscale instance. This requires that HTTPS is enabled in your Tailscale account and the Caddy process must either be running as root, or you must configure tailscaled to give your Caddy user permission to fetch certificates.

What I am trying:

• Caddy is running as root in same container as tailscaled.
• I am passing a *.ts.net domain
• The cert HTTPS tailscale process is working as tailscale cert is working as per How to know if Tailscale cert integration was attempted? #5041 (comment)

How do I get more log output on the attempt to work with the tailscale socket given the ts.net domain that's being used. I don't see any attempt in the logs even with DEBUG option.

[dcarrion87]

I've switched back to relying on the tailscale cert functionality and some hooks until I better understand the implicit *.ts.net voodoo that Caddy is doing in the background 😄

Thanks for your input.

[mholt]

@dcarrion87 Does it work outside of the container? I'm concerned that the odd container setup is probably breaking things.

How do I get more log output on the attempt to work with the tailscale socket given the ts.net domain that's being used. I don't see any attempt in the logs even with DEBUG option.

That's what all external certificate managers yielded no certificates and no errors {"sni": "REDACTED.REDACTED.ts.net"} means. The tailscale cert manager returned nil, nil which means the call to tscert.GetStatus() returned an error. Since the tailscale cert is implicit it treats that as "tailscale must not be installed or running, therefore skip it since it wasn't explicitly configured".

You can explicitly enable tailscale certs easily enough:

tls {
    get_certificate tailscale
}

And that should yield the specific error in the logs for you.

[francislavoie]

I think Caddy is looking specifically for this path:

https://github.com/tailscale/tscert/blob/54bbcb9f74e2a2a8660eb062baa9942ae12b0f19/internal/paths/paths.go#L30

[mholt]

I just added a debug log to the tailscale code as well, so if there's a (currently-hidden) error it'll now appear in the debug logs.

[dcarrion87]

@mholt bingo!

Added this with debugging:

tls {
    get_certificate tailscale
}

Revealed this:

ERROR   tls.handshake   getting certificate ... dial unix /var/run/tailscale/tailscaled.sock: connect: no such file or directory"}

Tailscale's default args for the tailscale container pins to /tmp/tailscaled.sock:

https://github.com/tailscale/tailscale/blob/65c24b6334e97d56b51157d31c8cf1a13ca26692/docs/k8s/run.sh#L23

After overriding this to the hard coded path caddy is looking for it all came good!

[mholt]

Great! Thanks for the follow-up.

I wonder if we should we make that socket path configurable?

I'm also kind of waiting on https://github.com/tailscale/caddy-tailscale. It is a Caddy plugin that, once the PR is merged, allows Caddy to join the tailnet and get TS certs without needing the TS daemon at all, is my understanding. So the need for that go away entirely soon.