Connection hangs/does not timeout after 350 seconds.

[mittalsuraj18]

When an application is running on an AWS VPC and using NAT gateway to communicate, there is a specific behavior in NAT gateway which is described here https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-troubleshooting.html as such

Internet connection drops after 350 seconds
Problem

Your instances can access the internet, but the connection drops after 350 seconds.
Cause

If a connection that's using a NAT gateway is idle for 350 seconds or more, the connection times out.

When a connection times out, a NAT gateway returns an RST packet to any resources behind the NAT gateway that attempt to continue the connection (it does not send a FIN packet).
Solution

To prevent the connection from being dropped, you can initiate more traffic over the connection. Alternatively, you can enable TCP keepalive on the instance with a value less than 350 seconds.

For example, if i have a single connection. and do something like,

client = AsyncClient(base_url=BASE_URL, http2=True)
_ = await client.get("/", timeout=1200)

The connection does not respond to RST packet. And times out after 1200 seconds instead of 350 seconds.

The behavior should have been either of the following

1. Timeout at 350 seconds
2. Periodically send keepalive packet to make sure NAT gateway does not timeout.

How i currently have solved it is by doing a periodic call every 60 seconds, which just calls a healtcheck endpoint of the same baseurl, to use the same connection.

[tomchristie]

Okay, it'd be helpful to understand if this behaviour is consistent across different clients...

• What is the behaviour for this case with httpx.Client().
• What is the behaviour for this case with httpx.AsyncClient() when running under asyncio. (You've answered this here.)
• What is the behaviour for this case with httpx.AsyncClient() when running under trio.
• What is the behaviour for this case with other clients... curl, requests, aiohttp.

Also can you show what the traceback is for the timeout here?
Presumably it's being raised here?

[grahamhar]

The same issue was raised here for requests and also here with a StackTrace, rather than going the route of introducing the package they suggested I took the following approach when hit with the same issue in requests with AWS, mine was transit gateway related rather than NAT Gateway.

import socket

from requests import Session
from urllib3.connection import HTTPConnection

HTTPConnection.default_socket_options = ( HTTPConnection.default_socket_options + [
        (socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1),
        (socket.SOL_TCP, socket.TCP_KEEPINTVL, 180)
        ]
)


session = Session()

# rest of codebase

Now I am looking to use https due to it's async capability I need to be able to achieve the sam settings.

[grahamhar]

Just looking through the codebase quickly as httpcore already supports setting socket options would we just need to expose this here?

[karpetrosyan]

We already support this. See #2635

[grahamhar]

@karpetrosyan thanks for the link I missed that when searching. I also couldn't find any reference in the httpx docs when searching for socket or TCPKeepAlive. I will try to do a PR to add this to the docs.

[tomchristie]

Related... this proposed(?) API would introduce a NetworkOptions config class and document these options more clearly. #3052