treq.testing.StubTreq: fix persisting twisted.web.server.Session objects between requests #328

bennr01 · 2021-04-13T13:28:28Z

This PR contains my proposed fix for #327. For details about the issue and this fix, see said issue.

tl;dr of original issue: Previously, a resource wrapped by treq.testing.StubTreq could not properly use request.getSession(), as it would always create a new twisted.web.server.Session instead of returning the same one. This is because treq.testing.StubTreq always create a new instance of twisted.web.server.Site for each request. As sessions are stored within a Site instance, this meant that upon completion of a request, the original session was lost. This PR solves this issue by moving the twisted.web.server.Site instance to an attribute of the agent. Additionally, the new treq.testing.StubTreq.cleanSessions() method provides a simple way of expiring the sessions in order to avoid leaving behind a dirty/unclean reactor. Although I am not sure if this is the best way to do this.

Before this commit, twisted.web.server.Session objects were not persisted properly between requests. Thus, two requests with the same cookies would result in two different sessions. See twisted#327 for details.

twm · 2021-04-14T05:13:04Z

src/treq/testing.py

+
+    def cleanSessions(self):
+        """
+        Clean up sessions to prevent leaving behind a dirty reactor.


The behavior of leaving a dirty reactor looks like a bug in Twisted to me, actually. The session's delayed call should be registered with the reactor passed to Site.

I filed https://twistedmatrix.com/trac/ticket/10177 for this issue.

I think that it could be worked around here by overriding the site's sessionFactory to pass the memory reactor to the Session constructor. Then you wouldn't need this method.

twm · 2021-04-14T05:15:55Z

src/treq/testing.py

+        Clean up sessions to prevent leaving behind a dirty reactor.
+        If you are using :obj:`StubTreq` with :obj:`twisted.web.server.Session`
+        objects, you most likely have to call this method once you are done,
+        for example during the tearDown of a unittest TestCase.


For stuff like this using self.addCleanup is almost always superior.

@twm Are you suggesting replacing this whole method to somehow use said function or to change the documentation?

To change the documentation, but it's a moot point if this method is going away.

twm · 2021-04-14T06:20:39Z

Thank you for the well-stated bug report and nicely polished PR! I think that the overall approach of reusing the Site is reasonable.

Would you find the cleanSessions method useful if it weren't necessary?

bennr01 · 2021-04-14T11:26:58Z

Hi,

Would you find the cleanSessions method useful if it weren't necessary?

It depends, but I'm leaning towards "no". It is useful when using treq.testing.StubTreq for testing twisted.web applications and wanting to reset sessions between requests, but there may be better ways to do that.
During debugging I encountered some weird cookie behavior which I am not sure is intentional (the same cookies being used between requests, even when no cookies parameter was passed to the request or the .clear() was called on the cookies), so maybe it's not necessary even when one wants to have a new session during the same test_* function.

Still, the primarily concern for writing .cleanSessions() was to provide a way to avoid having the user access private (as in 'starting with a underscore') attributes and methods. So if there would be a better way I'd be in favor of dropping .cleanSessions(). It's an explicit call for something that should be (IMO) implicitly cleaned up.

Personally, I also feel like .cleanSessions() is too specific for what it's supposed to do. A more generic method for completely resetting all associated states would be a more clean interface.

twm

Okay, then I'd say remove cleanSessions() entirely.

It is likely that we'll end up making some (all?) of these attributes public as part of #227. In practice tests that use StubTreq end up pretty tightly coupled to the details of these objects so I don't think it's useful to hide them — we couldn't make substantial internal changes without breaking folks' tests (even this PR has some Hyrum's Law risk in that regard, though I think it's acceptable).

Thanks!

bennr01 · 2021-04-21T10:59:57Z

Hi,

I have now removed .cleanSessions() again. Instead, I've moved the code directly into the test with a note to remove it once ticket 10177 is fixed.

twm

Getting closer! I've tried to clarify my prior feedback.

twm · 2021-04-26T03:33:13Z

src/treq/testing.py

@@ -88,6 +88,7 @@ def __init__(self, rootResource):
            reactor=self._memoryReactor,
            endpointFactory=_EndpointFactory(self._memoryReactor))
        self._rootResource = rootResource
+        self._serverFactory = Site(self._rootResource, reactor=self._memoryReactor)


I meant that you can cause the session to use the correct reactor by overriding sessionFactory here like this:

Suggested change

self._serverFactory = Site(self._rootResource, reactor=self._memoryReactor)

self._serverFactory = Site(self._rootResource, reactor=self._memoryReactor)

self._serverFactory.sessionFactory = lambda site, uid: Session(site, uid, reactor=self._memoryReactor)

Then you don't need the cleanup code in the test you added, and more importantly you don't need cleanup code in any test that uses RequestTraversalAgent.

twm · 2021-04-26T03:38:19Z

src/treq/test/test_testing.py

+        # or expiring the sessions
+
+        # manually expire the sessions.
+        for sid in list(stub._agent._serverFactory.sessions.keys()):


I think that you could do this without poking all these private variables by making _SessionIdTestResource append each session it creates to a public attribute and traversing that attribute here.

bennr01 · 2021-04-26T10:08:12Z

I've implemented the changes proposed above:

treq.testing.RequestTraversalAgent will now modify self._server.sessionFactory to always use the memory reactor (though seeing 10177: Sessions: use the site reactor, not the global one twisted#1587 this may have been uneccessary).
treq.test.test_testing._SessionIdTestResource will now keep track of the sessions it created and provide a method for expiring these sessions.

twm

Looks great! Thank you very much.

Sorry I lost track of this... I will merge now, and try to release by the weekend.

twm

Oh no, lint failure. :( Please fix that.

bennr01 · 2021-05-12T11:06:33Z

@twm Oops, I forgot that I configured the flake8 I'm using to ignore E501... Should be fixed now.

twm

Looks great! Thank you!

treq.testing.StubTreq: fix session persistence between requests

30a10bb

Before this commit, twisted.web.server.Session objects were not persisted properly between requests. Thus, two requests with the same cookies would result in two different sessions. See twisted#327 for details.

twm reviewed Apr 14, 2021

View reviewed changes

twm requested changes Apr 17, 2021

View reviewed changes

bennr01 and others added 2 commits April 21, 2021 12:56

remove .cleanSessions() again

f8b6e25

Merge branch 'master' into fix_327

00f1b82

twm mentioned this pull request Apr 26, 2021

10177: Sessions: use the site reactor, not the global one twisted/twisted#1587

Merged

6 tasks

twm requested changes Apr 26, 2021

View reviewed changes

treq.testing.RequestTraversalAgent: explicitly pass reactor to Site

c317256

twm approved these changes May 12, 2021

View reviewed changes

twm requested changes May 12, 2021

View reviewed changes

treq.testing: fix lint error (E501)

bbf412c

twm approved these changes May 13, 2021

View reviewed changes

twm merged commit b0c6172 into twisted:master May 13, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

treq.testing.StubTreq: fix persisting twisted.web.server.Session objects between requests #328

treq.testing.StubTreq: fix persisting twisted.web.server.Session objects between requests #328

bennr01 commented Apr 13, 2021

twm Apr 14, 2021

twm Apr 14, 2021

twm Apr 14, 2021

bennr01 Apr 14, 2021

twm Apr 17, 2021

twm commented Apr 14, 2021

bennr01 commented Apr 14, 2021

twm left a comment

bennr01 commented Apr 21, 2021

twm left a comment

twm Apr 26, 2021

twm Apr 26, 2021

bennr01 commented Apr 26, 2021

twm left a comment

twm left a comment

bennr01 commented May 12, 2021

twm left a comment

	self._serverFactory = Site(self._rootResource, reactor=self._memoryReactor)
	self._serverFactory = Site(self._rootResource, reactor=self._memoryReactor)
	self._serverFactory.sessionFactory = lambda site, uid: Session(site, uid, reactor=self._memoryReactor)

treq.testing.StubTreq: fix persisting twisted.web.server.Session objects between requests #328

treq.testing.StubTreq: fix persisting twisted.web.server.Session objects between requests #328

Conversation

bennr01 commented Apr 13, 2021

twm Apr 14, 2021

Choose a reason for hiding this comment

twm Apr 14, 2021

Choose a reason for hiding this comment

twm Apr 14, 2021

Choose a reason for hiding this comment

bennr01 Apr 14, 2021

Choose a reason for hiding this comment

twm Apr 17, 2021

Choose a reason for hiding this comment

twm commented Apr 14, 2021

bennr01 commented Apr 14, 2021

twm left a comment

Choose a reason for hiding this comment

bennr01 commented Apr 21, 2021

twm left a comment

Choose a reason for hiding this comment

twm Apr 26, 2021

Choose a reason for hiding this comment

twm Apr 26, 2021

Choose a reason for hiding this comment

bennr01 commented Apr 26, 2021

twm left a comment

Choose a reason for hiding this comment

twm left a comment

Choose a reason for hiding this comment

bennr01 commented May 12, 2021

twm left a comment

Choose a reason for hiding this comment