Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pass response_kw to HTTPConnectionPool through HTTPAdapter.send #6523

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Pass response_kw to HTTPConnectionPool through HTTPAdapter.send #6523

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f9b1eaf
Pass arguments from instance dict to urlopen in HTTPAdapter.send
joren485 Sep 9, 2023
fa37b4c
Add line of documentation about urllib3_response_options
joren485 Sep 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
37 changes: 24 additions & 13 deletions src/requests/adapters.py
View file
Open in desktop
Expand Up @@ -130,6 +130,7 @@ class HTTPAdapter(BaseAdapter):
"_pool_connections",
"_pool_maxsize",
"_pool_block",
"urllib3_response_options",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit; the values in here could be anything for the urlopen/_make_request methods. Defining them as response_options seems too tightly scoped. I left another comment below, but perhaps urllib3_urlopen_kwargs or just urllib3_send_kwargs would be more correct?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to be very clear this week only affect one thing because naming things too generically makes others and they should work for other things. Based on your other comments, I'm thinking we should add the word extra to the band to be clear these are only extra args

]

def __init__(
Expand All @@ -154,6 +155,8 @@ def __init__(

self.init_poolmanager(pool_connections, pool_maxsize, block=pool_block)

self.urllib3_response_options = {}
joren485 marked this conversation as resolved.
Show resolved Hide resolved

def __getstate__(self):
return {attr: getattr(self, attr, None) for attr in self.__attrs__}

Expand Down Expand Up @@ -435,6 +438,9 @@ def send(
):
"""Sends PreparedRequest object. Returns Response object.

It is possible to pass additional arguments to :meth:`urllib3.poolmanager.PoolManager.urlopen`
(e.g. `enforce_content_length`) by populating :attr:`HTTPAdapter.urllib3_response_options`.

:param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
:param stream: (optional) Whether to stream the request content.
:param timeout: (optional) How long to wait for the server to send
Expand Down Expand Up @@ -481,20 +487,25 @@ def send(
else:
timeout = TimeoutSauce(connect=timeout, read=timeout)

urlopen_kwargs = self.urllib3_response_options.copy()
urlopen_kwargs.update(
{
"method": request.method,
"url": url,
"body": request.body,
"headers": request.headers,
"redirect": False,
"assert_same_host": False,
"preload_content": False,
"decode_content": False,
"retries": self.max_retries,
"timeout": timeout,
"chunked": chunked,
}
)

Comment on lines +490 to +506
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering if this should be done in the opposite order (defaults updated by self.urllib3_response_options). If we add this feature, we'll almost certainly get future issues with questions like "Why can't I change preload_content or decode_content with the <variable name>?".

If we're going to expose something like this to avoid custom send implementations, it should probably be the full set of arguments. We can still keep the same defaults, but whatever users want to pass us, we should just forward.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my experience if there's something like this designed to help people access something as an escape hatch it will be misunderstood and misused. Copying the dictionary and updating it will avoid footguns. No one will override the headers, method, data, or anything else. I'm not particularly strongly invested in this feature as a whole but the other way I think we can enable this (because overriding send is a nightmare) is breaking out the actual call to urlopen into a different method that can be overridden.

try:
resp = conn.urlopen(
method=request.method,
url=url,
body=request.body,
headers=request.headers,
redirect=False,
assert_same_host=False,
preload_content=False,
decode_content=False,
retries=self.max_retries,
timeout=timeout,
chunked=chunked,
)
resp = conn.urlopen(**urlopen_kwargs)

except (ProtocolError, OSError) as err:
raise ConnectionError(err, request=request)
Expand Down