Hmmm, thanks for the report. I'll try to look into this soon.

What is in your logs before, during, and after a config change?

I would expect the server socket to shut down after the grace period and sever existing connections to clients.

As for the dynamic upstreams, they are global and persist through config reloads, but keyed by the SRV address, so if those change the old ones should no longer be used I think.

I've attached a dump of the logs log.gz, I started Caddy with

docker run --rm --network host --privileged --cap-add NET_ADMIN -v /root/config/Caddyfile:/etc/caddy/Caddyfile -v /root/config/data:/data caddy:2.7.6-alpine > log 2> error_log

Accessed a page on the site, then entered the container, modified the Caddyfile and did caddy reload --config /etc/caddy/Caddyfile

The initial service record was php-pool-dda10003a747409ee137207837a8eae7.service.consul and I changed it to xxx-php-pool-dda10003a747409ee137207837a8eae7.service.consul

Reload happened at 2024/03/28 03:58:58.357

Old record was used until 2024/03/28 04:00:04.674, at that point it failed as in this case the upstream doesn't exist, but it doesn't matter if the upstream exists or not. Sometimes it'll hold it longer, but usually 2 - 3 minutes.

Caddyfile used:

{
  admin localhost:2019

  log {
    output stdout
    level DEBUG
    format console
  }

  shutdown_delay 1s
  grace_period 3s
  debug
}

# Snippets

(reverseProxyConfig) {
  header_up +X_FORWARDED_PORT 443
  lb_policy least_conn
  lb_try_duration 5s
  lb_try_interval 250ms
  fail_duration 2s
  unhealthy_status 5xx

	health_uri /
	health_interval 10s
	health_timeout 5s
	health_status 5xx
}

(logsConfig) {
  log {
    output stdout
    level DEBUG
    format console
  }
}

# Health Check
:8080 {
  respond /health-check OK 200
}

ingresstest.fortix.systems {
  import logsConfig
  encode zstd gzip

    reverse_proxy {
      dynamic srv {
        name php-pool-dda10003a747409ee137207837a8eae7.service.consul
        refresh 10s
        dial_timeout 5s
      }
      import reverseProxyConfig
      transport http {
        versions 2
        keepalive off
	keepalive_interval 5s
	keepalive_idle_conns 2
        read_buffer 16KiB
        write_buffer 16KiB

      }
  }

}

Thanks

From the log I found out all the requests are HTTP/3.0. It's due to old h3 connections being active after configuration reload. I'll see what I can do with quic-go, which caddy depends for http3.

@paularlott Can you try xcaddy build close-h3-connections to see if it's fixed?

@WeidiDeng I've only had a chance for a quick test so far, but it's looking like that resolves the issues, thank you.

@WeidiDeng You are a WIZARD 🎉 🧙‍♂️ can't wait to review this 😃

@paularlott Do you mind giving this patch a go? This properly closes http3 servers without interrupting ongoing requests unless a grace period is reached.

@WeidiDeng I've just built it and deployed it as the ingress controller for some test sites and it seems to be following the changes to the upstream SRV records as it should. I'll leave it running so we can give it a good test.

Thank you