Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add streaming multipart/form-data encoder #3361

Draft
wants to merge 21 commits into
base: main
Choose a base branch
from

Commits on Jan 2, 2022

  1. Add multipart/form-data streaming encoder

    I initially implemented this for Requests in the requests-toolbelt but
    this was already pretty generic because Requests just passes file-like
    objects (which is how the streaming encoder behaves) directly to
    urllib3. All that needed to change was what we were relying on from the
    requests namespace and imports and such.
    
    This also adds the decoder in ther same breath because it's easier to
    ensure that's all working together properly in one and it all fits
    together nicely.
    
    One thing we _could_ do is consolidate a bunch of the logic too and make
    `encode_multipart_formdata` rely on the streaming encoder and call
    `getall` instead so that we don't have 2 implementations of the same
    logic.
    sigmavirus24 committed Jan 2, 2022
    Configuration menu
    Copy the full SHA
    c8c153e View commit details
    Browse the repository at this point in the history

Commits on Jun 4, 2022

  1. Configuration menu
    Copy the full SHA
    6c6117c View commit details
    Browse the repository at this point in the history
  2. Fix mypy

    Type annotations made it clear that a dangling comma was added by
    mistake, converting a value into a tuple.
    pquentin committed Jun 4, 2022
    Configuration menu
    Copy the full SHA
    b2f8061 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2022

  1. Measure test coverage

    pquentin committed Jun 12, 2022
    Configuration menu
    Copy the full SHA
    ba64375 View commit details
    Browse the repository at this point in the history

Commits on Aug 19, 2022

  1. Configuration menu
    Copy the full SHA
    c996550 View commit details
    Browse the repository at this point in the history
  2. Revert back to using Union[str, bytes] for rfc2231

    It's incorrect in the context of the streaming encoder changes.
    pquentin committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    2ecc96d View commit details
    Browse the repository at this point in the history
  3. Fix docs generation

    pquentin committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    06ba311 View commit details
    Browse the repository at this point in the history

Commits on Mar 11, 2024

  1. Merge remote-tracking branch 'origin/main' into add-multipart-formdat…

    …a-encoder-2024
    
    # Conflicts:
    #	docs/conf.py
    #	src/urllib3/connection.py
    #	src/urllib3/fields.py
    #	src/urllib3/filepost.py
    #	test/with_dummyserver/test_connectionpool.py
    alexwlchan committed Mar 11, 2024
    Configuration menu
    Copy the full SHA
    be4633c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    591b0c3 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    4329d0f View commit details
    Browse the repository at this point in the history
  4. Ensure we close the open files at the end of the test

    Previously this test was failing with a warning that was being raised
    as an error:
    
    _____________________________________________ TestMultipartEncoder.test_regression_1 ______________________________________________
    Traceback (most recent call last):
      File "/Users/alexwlchan/repos/urllib3/.nox/test-3-12/lib/python3.12/site-packages/pluggy/_callers.py", line 155, in _multicall
        teardown[0].send(outcome)
      File "/Users/alexwlchan/repos/urllib3/.nox/test-3-12/lib/python3.12/site-packages/_pytest/unraisableexception.py", line 88, in pytest_runtest_call
        yield from unraisable_exception_runtest_hook()
      File "/Users/alexwlchan/repos/urllib3/.nox/test-3-12/lib/python3.12/site-packages/_pytest/unraisableexception.py", line 78, in unraisable_exception_runtest_hook
        warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
    pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
    
    Traceback (most recent call last):
      File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/unittest/case.py", line 589, in _callTestMethod
        if method() is not None:
           ^^^^^^^^
    ResourceWarning: unclosed file <_io.BufferedReader name='/Users/alexwlchan/repos/urllib3/test/multipart/test_encoder.py'>
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/Users/alexwlchan/repos/urllib3/.nox/test-3-12/lib/python3.12/site-packages/_pytest/runner.py", line 341, in from_call
        result: Optional[TResult] = func()
                                    ^^^^^^
      File "/Users/alexwlchan/repos/urllib3/.nox/test-3-12/lib/python3.12/site-packages/_pytest/runner.py", line 262, in <lambda>
        lambda: ihook(item=item, **kwds), when=when, reraise=reraise
                ^^^^^^^^^^^^^^^^^^^^^^^^
      File "/Users/alexwlchan/repos/urllib3/.nox/test-3-12/lib/python3.12/site-packages/pluggy/_hooks.py", line 501, in __call__
        return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/Users/alexwlchan/repos/urllib3/.nox/test-3-12/lib/python3.12/site-packages/pluggy/_manager.py", line 119, in _hookexec
        return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/Users/alexwlchan/repos/urllib3/.nox/test-3-12/lib/python3.12/site-packages/pluggy/_callers.py", line 159, in _multicall
        _warn_teardown_exception(hook_name, teardown[1], e)
      File "/Users/alexwlchan/repos/urllib3/.nox/test-3-12/lib/python3.12/site-packages/pluggy/_callers.py", line 49, in _warn_teardown_exception
        warnings.warn(PluggyTeardownRaisedWarning(msg), stacklevel=5)
    pluggy.PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
    Plugin: unraisableexception, Hook: pytest_runtest_call
    PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
    
    Traceback (most recent call last):
      File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/unittest/case.py", line 589, in _callTestMethod
        if method() is not None:
           ^^^^^^^^
    ResourceWarning: unclosed file <_io.BufferedReader name='/Users/alexwlchan/repos/urllib3/test/multipart/test_encoder.py'>
    alexwlchan committed Mar 11, 2024
    Configuration menu
    Copy the full SHA
    84c1773 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    8dc4245 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    786613e View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    40ea533 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    5dc22c2 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    0814a9a View commit details
    Browse the repository at this point in the history

Commits on Mar 16, 2024

  1. Configuration menu
    Copy the full SHA
    d14f34d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e0d588f View commit details
    Browse the repository at this point in the history

Commits on Mar 24, 2024

  1. Configuration menu
    Copy the full SHA
    b0be345 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    6a8a3a6 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    581f906 View commit details
    Browse the repository at this point in the history