Duplicate requests in latest_requests() if there are chunks

[VelorumS]

Was fine with 1.0.5, does duplicates in 1.1.0.

The change: b6161ce#r50812634

-            cls.latest_requests[-1] = request
+            pos = cls.latest_requests.index(request)
+            cls.latest_requests[pos] = request

httpretty.historify_request() is called multiple times for the same request.

From here:

  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.6/http/client.py", line 1281, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1327, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1276, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1081, in _send_output
    self.send(chunk)
  File "/usr/lib/python3.6/http/client.py", line 1002, in send
    self.sock.sendall(data)
  File "/usr/local/lib/python3.6/dist-packages/httpretty/core.py", line 683, in sendall

And here:

  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.25.1-py3.6.egg/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.6/http/client.py", line 1281, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1327, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1276, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1042, in _send_output
    self.send(msg)
  File "/usr/lib/python3.6/http/client.py", line 1002, in send
    self.sock.sendall(data)
  File "/usr/local/lib/python3.6/dist-packages/httpretty/core.py", line 699, in sendall

[ThiefMaster]

These duplicate requests broke my CI. Pinning httpretty<1.1 is a workaround but a fix would be appreciated.

[Gidgidonihah]

I also came looking for this because of a broken CI.

[gabrielfalcao]

Oh gee, thanks @ChipmunkV for the solution. I'll get a new release out soon and will ping you folks once it's done.

[gabrielfalcao]

@ChipmunkV, @Gidgidonihah , @ThiefMaster I'm attempting to write a functional test to reproduce the bug.
Would it be possible for you to share some example code to reproduce the issue?

[Gidgidonihah]

FYI, I cloned master after merge and tested it, and my tests are still failing with the wrong number of requests in latest_requests. I don't yet have a MWE example, but I would love to get one worked up.

[gabrielfalcao]

@Gidgidonihah I'd love to work with you in a solution for this, I just want to make sure there is an automated test to reproduce the issue.

I tried running the tests from indico-plugins-cern that @ThiefMaster mentioned but ran into some issues and ended up spending too much time, so I gave up on running the test.
I also had trouble understanding what these tests are really trying to accomplish: https://github.com/indico/indico-plugins-cern/blob/03acd644ab2476b37c693250f4ea62b9f26985c1/ravem/tests/operations/connect_test.py#L62-L70

My main concern with httpretty.latest_requests() is whether it have been storing requests incorrectly up to the release 1.0.5. I that is the case, then I'm afraid that you, @ThiefMaster and anyone else who relies on len(httpretty.latest_requests()) might have been relying on an HTTPretty bug rather than a feature.

I take automated testing very seriously and want to make sure that whatever the case is, we can find the best possible solution for HTTPretty and ensure that your tests pass correctly (without false-positives).

How do you think I should proceed with this issue?

[gabrielfalcao]

The issue title suggests that the problem occurs "if there are chunks". I suppose this is a problem with chunked requests, is that the case for everyone here?
Would any of you help me reproducing the issue?