New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Issue 408 fix, Memory leak was generated from Event #524
base: master
Are you sure you want to change the base?
Conversation
changes event.py: - self._result and self._exc should not be class's static attributes - self._exc should not store exception trace, instead if exception occurred True - self._exc do not need to store exception trace, the same exception is called with the schedule_call_global for self._waiters timers - no need to throw self._exc on wait, it is thrown on the switch changes greenthread.py: (adjustments) - use of super for init inherited class - main method, try send, except send_exception and raise, and finally _resolve_links Signed-off-by: Kashirin Alex <kashirin.alex@gmail.com>
added test_no_mem_leaks to event_test.py added tox.ini dependencies objgraph==3.4.0 Signed-off-by: Kashirin Alex <kashirin.alex@gmail.com>
changes greenthread.py: (adjustments) - use of super for init inherited class Signed-off-by: Kashirin Alex <kashirin.alex@gmail.com>
changes event.py: self._result and self._exc should not be class's static attributes (the Values were passing between the different Event instances) self._exc is set back to None on last waiter has it's results (references kept them self, causing GC not to collect, Event has _exc and _exc has trace frames object in touched) Signed-off-by: Kashirin Alex <kashirin.alex@gmail.com>
- added test_no_mem_leaks to event_test.py - added tox.ini dependencies objgraph==3.4.0 Signed-off-by: Kashirin Alex <kashirin.alex@gmail.com>
changes greenthread.py: (adjustments) - use of super for init inherited class - main method, _resolve_links finally Signed-off-by: Kashirin Alex <kashirin.alex@gmail.com>
changes event.py: - self._result and self._exc should not be class's static attributes Signed-off-by: Kashirin Alex <kashirin.alex@gmail.com>
…into 408-event-memleak Signed-off-by: Kashirin Alex <kashirin.alex@gmail.com> # Conflicts: # eventlet/event.py # eventlet/greenthread.py # tests/event_test.py
changes event.py: - self._exc should not store exception trace, instead if exception occurred self._exc=True (references kept them self, causing GC not to collect, Event has _exc and _exc has trace frames object in touched) - self._exc do not need to store exception trace, the same exception is called with the schedule_call_global for self._waiters timers - a wait on a triggered event is returning _results (without throw) None is exception/timeout Signed-off-by: Kashirin Alex <kashirin.alex@gmail.com>
changes event.py: - self._exc should not store exception trace, instead if exception occurred self._exc=True (references kept them self, causing GC not to collect, Event has _exc and _exc has trace frames object in touched) - self._exc do not need to store exception trace, the same exception is called with the schedule_call_global for self._waiters timers - a wait on a triggered event with sellf._exc will throw current (GreenletExit) Signed-off-by: Kashirin Alex <kashirin.alex@gmail.com>
changes event.py: - self._exc should not store exception trace, instead if exception occurred self._exc=True (references kept them self, causing GC not to collect, Event has _exc and _exc has trace frames object in touched) Signed-off-by: Kashirin Alex <kashirin.alex@gmail.com>
Sorry, Unable to find the Fix. |
Signed-off-by: Kashirin Alex <kashirin.alex@gmail.com>
…r test use Signed-off-by: Kashirin Alex <kashirin.alex@gmail.com>
…r test use Signed-off-by: Kashirin Alex <kashirin.alex@gmail.com>
…r test use Signed-off-by: Kashirin Alex <kashirin.alex@gmail.com>
…r test use Signed-off-by: Kashirin Alex <kashirin.alex@gmail.com>
Signed-off-by: Kashirin Alex <kashirin.alex@gmail.com>
Signed-off-by: Kashirin Alex <kashirin.alex@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's redundant to list the changes being made in commit messages.
Instead an explanation of why things are being done is needed. After this pull request is merged the repository needs to contain a helpful commit message explaining that.
Additionally, if you don't have a solution ready please consider testing it locally as creating a bunch of commits that fix each other floods inboxes of people watching the repository.
|
||
def __init__(self): | ||
self._result = None | ||
self._exc = None |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems unrelated to the issue at hand.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
At this point it is the only thing that can cause a one not garbage collected object.
@@ -167,7 +167,7 @@ class GreenThread(greenlet.greenlet): | |||
""" | |||
|
|||
def __init__(self, parent): | |||
greenlet.greenlet.__init__(self, self.main, parent) | |||
super(GreenThread, self).__init__(self.main, parent) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also seems unrelated to the issue at hand
|
||
import objgraph | ||
import gc | ||
import sys |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Those can be imported at the module level just fine, why import here?
while gc.collect(): | ||
pass | ||
|
||
eventlet.hubs.get_hub().set_timer_exceptions(True) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have to make sure this is executed even if this test fails earlier on.
@@ -67,6 +67,7 @@ deps = | |||
py{27,34}-{selects,poll,epolls}: pyopenssl==17.3.0 | |||
setuptools==38.5.1 | |||
{selects,poll,epolls}: pyzmq==17.0.0 | |||
objgraph==3.4.0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is an unnecessarily heavy dependency for what's being done here, we can count the amount of objects of a specific type just fine without it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will be great to know which method can replace the objgraph.count('eventlet.greenthread.GreenThread')
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
>>> import gc
>>> import objgraph
>>>
>>> class X:
2 pass
>>>
>>> objgraph.count('X')
0
>>> sum(1 for o in gc.get_objects() if isinstance(o, X))
0
>>> x = X()
>>> objgraph.count('X')
1
>>> sum(1 for o in gc.get_objects() if isinstance(o, X))
1
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, thanks,
I'll keep it updated with a solution to overcome/change the use of self._exc (if found of-course)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it might should be a method of the tests/init.py, and it has already gc imported
def check_ref_count(Instance, expected=0):
c = sum(1 for o in gc.get_objects() if isinstance(o, Instance))
assert c == expected, \
"%s ref expected: %d counted: %d " % (Instance, expected, c)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
or it can be better with a with statement
def check_ref_count(Instance, expected=0):
try:
pre_c = sum(1 for o in gc.get_objects() if isinstance(o, Instance))
finally:
c = sum(1 for o in gc.get_objects() if isinstance(o, Instance))
assert c-pre_c == expected, \
"%s ref pre-counted: %d expected: %d \ counted: %d "
% (Instance, pre_c, expected, c)
and work with:
with check_ref_count(Instance):
#code
Yes, the follow-up commits came out of nowhere, as there was more to the issue and initial checks were not sufficient although no issues have been seen for a while now (different type of occurrences/usages)
|
#408, fix for MemLeak on event exceptions …
(good with PyPy, while issues with Python 2/3)
changes event.py:
changes greenthread.py: (adjustments)
#408, Test for fix for MemLeak on event exceptions …