Duplicate requests in latest_requests() if there are chunks #425

VelorumS · 2021-05-14T09:57:17Z

Was fine with 1.0.5, does duplicates in 1.1.0.

The change: b6161ce#r50812634

-            cls.latest_requests[-1] = request
+            pos = cls.latest_requests.index(request)
+            cls.latest_requests[pos] = request

httpretty.historify_request() is called multiple times for the same request.

From here:

  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.6/http/client.py", line 1281, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1327, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1276, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1081, in _send_output
    self.send(chunk)
  File "/usr/lib/python3.6/http/client.py", line 1002, in send
    self.sock.sendall(data)
  File "/usr/local/lib/python3.6/dist-packages/httpretty/core.py", line 683, in sendall

And here:

  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.6/http/client.py", line 1281, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1327, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1276, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1042, in _send_output
    self.send(msg)
  File "/usr/lib/python3.6/http/client.py", line 1002, in send
    self.sock.sendall(data)
  File "/usr/local/lib/python3.6/dist-packages/httpretty/core.py", line 699, in sendall

The text was updated successfully, but these errors were encountered:

Feature: Display mismatched URL within UnmockedError whenever possible. Closes #388 Feature: Display mismatched URL via logging. #419 Add new properties to `httpretty.core.HTTPrettyRequest` (protocol, host, url, path, method).

ThiefMaster · 2021-05-19T12:24:05Z

These duplicate requests broke my CI. Pinning httpretty<1.1 is a workaround but a fix would be appreciated.

See gabrielfalcao/HTTPretty#425

Gidgidonihah · 2021-05-19T20:49:35Z

I also came looking for this because of a broken CI.

gabrielfalcao · 2021-05-19T21:57:38Z

Oh gee, thanks @ChipmunkV for the solution. I'll get a new release out soon and will ping you folks once it's done.

gabrielfalcao · 2021-05-20T18:09:39Z

@ChipmunkV, @Gidgidonihah , @ThiefMaster I'm attempting to write a functional test to reproduce the bug.
Would it be possible for you to share some example code to reproduce the issue?

* attempt to reproduce #425 * 💅

Gidgidonihah · 2021-05-24T23:05:13Z

FYI, I cloned master after merge and tested it, and my tests are still failing with the wrong number of requests in latest_requests. I don't yet have a MWE example, but I would love to get one worked up.

gabrielfalcao · 2021-05-25T15:09:33Z

@Gidgidonihah I'd love to work with you in a solution for this, I just want to make sure there is an automated test to reproduce the issue.

I tried running the tests from indico-plugins-cern that @ThiefMaster mentioned but ran into some issues and ended up spending too much time, so I gave up on running the test.
I also had trouble understanding what these tests are really trying to accomplish: https://github.com/indico/indico-plugins-cern/blob/03acd644ab2476b37c693250f4ea62b9f26985c1/ravem/tests/operations/connect_test.py#L62-L70

My main concern with httpretty.latest_requests() is whether it have been storing requests incorrectly up to the release 1.0.5. I that is the case, then I'm afraid that you, @ThiefMaster and anyone else who relies on len(httpretty.latest_requests()) might have been relying on an HTTPretty bug rather than a feature.

I take automated testing very seriously and want to make sure that whatever the case is, we can find the best possible solution for HTTPretty and ensure that your tests pass correctly (without false-positives).

How do you think I should proceed with this issue?

gabrielfalcao · 2021-05-25T15:12:08Z

The issue title suggests that the problem occurs "if there are chunks". I suppose this is a problem with chunked requests, is that the case for everyone here?
Would any of you help me reproducing the issue?

ThiefMaster · 2021-05-25T15:15:28Z

Feel free to send a PR against my repo that reverts this commit and installs httpretty from git.

That way you don't need to setup the somewhat complex environment locally but still see if the error happens or not.

Alternatively, if you point me to the branch from which I can install the (potentially) fixed version I can give it a try locally.

gabrielfalcao · 2021-05-25T15:28:59Z

Thanks @ThiefMaster, I already merged 0eb4b16 and released the changes in version 1.1.3 if you could kindly test it :)

gabrielfalcao · 2021-05-25T15:29:38Z

@Gidgidonihah if you could also test your code against 1.1.3 that would be great.
Thanks folks!

Gidgidonihah · 2021-05-25T19:43:02Z

@gabrielfalcao Same result for me. Thanks for getting on this quickly. I'll try to get an MWE together at the end of my sprint.

Gidgidonihah · 2021-05-27T18:12:56Z

@gabrielfalcao Here's a gist for a MWE.

Using a python 3.9.5 virtualenv, just ran:

pip install -r requirements.txt
python -m unittest

tim-schilling · 2021-05-27T19:02:09Z

@gabrielfalcao It looks like there's another thing duplicating requests (I think). I've created a unit test within the repo that fails.

I narrowed it down to a POST that has data. Unless it's a preflight request that's being stored? That would explain the body being 0, though I'd expect the method to be OPTIONS and not POST.

daveygit2050 · 2021-06-22T09:51:32Z

@gabrielfalcao I can confirm that it only appears to impact POST requests as @tim-schilling suggested:

import httpretty
import requests

httpretty.enable()

httpretty.register_uri(method=httpretty.POST, uri="https://example.com/foo")
requests.post("https://example.com/foo", data="bar")
len(httpretty.latest_requests())
# returns 2

httpretty.reset()

httpretty.register_uri(method=httpretty.GET, uri="https://example.com/foo")
requests.get("https://example.com/foo")
len(httpretty.latest_requests())
# returns 1

gabrielfalcao · 2021-06-22T12:39:46Z

@gabrielfalcao I can confirm that it only appears to impact POST requests as @tim-schilling suggested:

import httpretty
import requests

httpretty.enable()

httpretty.register_uri(method=httpretty.POST, uri="https://example.com/foo")
requests.post("https://example.com/foo", data="bar")
len(httpretty.latest_requests())
# returns 2

httpretty.reset()

httpretty.register_uri(method=httpretty.GET, uri="https://example.com/foo")
requests.get("https://example.com/foo")
len(httpretty.latest_requests())
# returns 1

Thanks @daveygit2050 I'll incorporate this snippet into a new test case to reproduce the issue as soon as I can manage. Unfortunately I might not be able to attend to this until next week but I'll keep yall posted.

ThiefMaster · 2021-10-27T12:32:46Z

A release containing this fix would be great!

enthooz · 2021-11-02T16:18:12Z

Thank you for creating a tool that helps improve automated tests! 🤗

I recently encountered this issue and wanted to add more info. In my testing, this applies to any requests (POST, PUT, DELETE, and/or PATCH) when they contain a request body. Below is a test file that demonstrates this behavior. (I did not include tests with request bodies for HTTP methods which do not allow request bodies (GET, HEAD, CONNECT, and OPTIONS), however these also demonstrate the behavior when a request body is provided.)

import httpretty
import urllib.request as request


class TestHTTpretty:
    @httpretty.activate(verbose=True, allow_net_connect=False)
    def run(self, method, data=None):
        url = 'https://example.com/foo'
        httpretty.register_uri(method=method, uri=url)
        req = request.Request(url=url, method=method, data=data)
        request.urlopen(req)
        assert len(httpretty.latest_requests()) == 1

    # the first eight tests with empty request bodies succeed
    def test_get(self):
        self.run(httpretty.GET)

    def test_head(self):
        self.run(httpretty.HEAD)

    def test_connect(self):
        self.run(httpretty.CONNECT)

    def test_options(self):
        self.run(httpretty.OPTIONS)

    def test_post_empty(self):
        self.run(httpretty.POST)

    def test_put_empty(self):
        self.run(httpretty.PUT)

    def test_delete_empty(self):
        self.run(httpretty.DELETE)

    def test_patch_empty(self):
        self.run(httpretty.PATCH)

    # the following four tests with populated request bodies fail
    def test_post_with_data(self):
        self.run(httpretty.POST, b'lorem ipsum')

    def test_put_with_data(self):
        self.run(httpretty.PUT, b'lorem ipsum')

    def test_delete_with_data(self):
        self.run(httpretty.DELETE, b'lorem ipsum')

    def test_patch_with_data(self):
        self.run(httpretty.PATCH, b'lorem ipsum')

Output:

============================ short test summary info ============================
FAILED test/httpretty_test.py::TestHTTpretty::test_post_with_data - assert 2 == 1
FAILED test/httpretty_test.py::TestHTTpretty::test_put_with_data - assert 2 == 1
FAILED test/httpretty_test.py::TestHTTpretty::test_delete_with_data - assert 2 == 1
FAILED test/httpretty_test.py::TestHTTpretty::test_patch_with_data - assert 2 == 1

cansin · 2022-03-03T08:09:45Z

I am dealing with the exact same issue. Looks like there was a fix merged into the master. As @ThiefMaster was pointing out, @gabrielfalcao would it be possible to deploy a new release?

This commit updates some packages to be same as in other GrimoireLab repsoitories. httpretty>1.1 contains a bug which causing fail of our test. For more info see: gabrielfalcao/HTTPretty#425 Include setuptools because is a dependency of uri but it is not explicit and latests version of Poetry removes setuptools. Signed-off-by: Jose Javier Merchante <jjmerchante@bitergia.com>

chassing · 2022-11-29T10:27:17Z

+1 for a 1.1.5 release

This commit updates some packages to be same as in other GrimoireLab repsoitories. httpretty>1.1 contains a bug which causing fail of our test. For more info see: gabrielfalcao/HTTPretty#425 Include setuptools because is a dependency of uri but it is not explicit and latests version of Poetry removes setuptools. Signed-off-by: Jose Javier Merchante <jjmerchante@bitergia.com>

ThiefMaster · 2023-07-05T13:54:42Z

@gabrielfalcao Can we please get an 1.1.5 release containing this fix?

gabrielfalcao · 2023-07-06T01:59:24Z

Salutations all, my macbook got damaged and I'm quite a bit limited at present due to the time taken for the new parts to arrive. I plan on applying the patch and resolve the situation as soon as the universe collaborates. Please hang in there and I'll report back as soon as humanely possible. -- Kind regards, Gabriel

gabrielfalcao · 2023-07-07T21:38:17Z

Hi folks,
I presently find myself somewhat able to work on this issue. I'll do what I can to make a release. Thanks for your patience so far.

chassing · 2023-12-07T10:05:09Z

Hi @gabrielfalcao,

This is a friendly reminder; I hope you'll find some time to make a new release :)

Thank you

sfuhrm · 2024-01-03T09:24:20Z

Problem exists in httpretty==1.1.4
httpretty==1.0.5 works as a workaround

@gabrielfalcao thanks for your great effort, it is appreciated!

ThiefMaster added a commit to indico/indico-plugins-cern that referenced this issue May 19, 2021

Pin httpretty (fix ravem tests)

cc23b58

See gabrielfalcao/HTTPretty#425

gabrielfalcao self-assigned this May 19, 2021

gabrielfalcao added a commit that referenced this issue May 20, 2021

attempt to reproduce #425

03b74fb

gabrielfalcao mentioned this issue May 20, 2021

Fix #425 #429

Merged

gabrielfalcao closed this as completed in #429 May 23, 2021

gabrielfalcao added a commit that referenced this issue May 23, 2021

Fix #425 (#429)

0eb4b16

* attempt to reproduce #425 * 💅

gabrielfalcao reopened this May 25, 2021

gabrielfalcao closed this as completed in c255165 Sep 13, 2021

gabrielfalcao reopened this Oct 28, 2021

bednar mentioned this issue Jul 22, 2022

chore(deps-dev): bump httpretty from 1.0.5 to 1.1.4 influxdata/influxdb-client-python#470

Closed

fr12k mentioned this issue Sep 14, 2023

Document about the ways to write unit tests for Bolt apps slackapi/bolt-python#380

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Duplicate requests in latest_requests() if there are chunks #425

Duplicate requests in latest_requests() if there are chunks #425

VelorumS commented May 14, 2021 •

edited

ThiefMaster commented May 19, 2021 •

edited

Gidgidonihah commented May 19, 2021

gabrielfalcao commented May 19, 2021

gabrielfalcao commented May 20, 2021 •

edited

Gidgidonihah commented May 24, 2021

gabrielfalcao commented May 25, 2021

gabrielfalcao commented May 25, 2021

ThiefMaster commented May 25, 2021 •

edited

gabrielfalcao commented May 25, 2021

gabrielfalcao commented May 25, 2021

Gidgidonihah commented May 25, 2021

Gidgidonihah commented May 27, 2021

tim-schilling commented May 27, 2021

daveygit2050 commented Jun 22, 2021 •

edited

gabrielfalcao commented Jun 22, 2021

ThiefMaster commented Oct 27, 2021

enthooz commented Nov 2, 2021

cansin commented Mar 3, 2022

chassing commented Nov 29, 2022

ThiefMaster commented Jul 5, 2023

gabrielfalcao commented Jul 6, 2023 via email

gabrielfalcao commented Jul 7, 2023

chassing commented Dec 7, 2023

sfuhrm commented Jan 3, 2024

Duplicate requests in latest_requests() if there are chunks #425

Duplicate requests in latest_requests() if there are chunks #425

Comments

VelorumS commented May 14, 2021 • edited

ThiefMaster commented May 19, 2021 • edited

Gidgidonihah commented May 19, 2021

gabrielfalcao commented May 19, 2021

gabrielfalcao commented May 20, 2021 • edited

Gidgidonihah commented May 24, 2021

gabrielfalcao commented May 25, 2021

gabrielfalcao commented May 25, 2021

ThiefMaster commented May 25, 2021 • edited

gabrielfalcao commented May 25, 2021

gabrielfalcao commented May 25, 2021

Gidgidonihah commented May 25, 2021

Gidgidonihah commented May 27, 2021

tim-schilling commented May 27, 2021

daveygit2050 commented Jun 22, 2021 • edited

gabrielfalcao commented Jun 22, 2021

ThiefMaster commented Oct 27, 2021

enthooz commented Nov 2, 2021

cansin commented Mar 3, 2022

chassing commented Nov 29, 2022

ThiefMaster commented Jul 5, 2023

gabrielfalcao commented Jul 6, 2023 via email

gabrielfalcao commented Jul 7, 2023

chassing commented Dec 7, 2023

sfuhrm commented Jan 3, 2024

VelorumS commented May 14, 2021 •

edited

ThiefMaster commented May 19, 2021 •

edited

gabrielfalcao commented May 20, 2021 •

edited

ThiefMaster commented May 25, 2021 •

edited

daveygit2050 commented Jun 22, 2021 •

edited