Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix incorrect timeout warnings in AWS Lambda and GCP integrations #854

Merged
merged 6 commits into from Oct 14, 2020
Merged

Fix incorrect timeout warnings in AWS Lambda and GCP integrations #854

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
d808d64
Changes:
Oct 12, 2020
33eecb6
Changes:
Oct 12, 2020
1806fef
Changes:
Oct 13, 2020
7e841a5
Changes:
Oct 14, 2020
80d79aa
ref: reduce indentation
rhcarvalho Oct 14, 2020
59b32d4
fix: remove trailing whitespace
rhcarvalho Oct 14, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
8 changes: 7 additions & 1 deletion sentry_sdk/integrations/aws_lambda.py
View file
Open in desktop
Expand Up @@ -83,6 +83,8 @@ def sentry_handler(event, context, *args, **kwargs):
_make_request_event_processor(event, context, configured_time)
)
scope.set_tag("aws_region", context.invoked_function_arn.split(":")[3])

timeout_thread = None
# Starting the Timeout thread only if the configured time is greater than Timeout warning
# buffer and timeout_warning parameter is set True.
if (
Expand All @@ -94,7 +96,8 @@ def sentry_handler(event, context, *args, **kwargs):
) / MILLIS_TO_SECONDS

timeout_thread = TimeoutThread(
waiting_time, configured_time / MILLIS_TO_SECONDS
waiting_time,
configured_time / MILLIS_TO_SECONDS,
)

# Starting the thread to raise timeout warning exception
Expand All @@ -116,6 +119,9 @@ def sentry_handler(event, context, *args, **kwargs):
)
hub.capture_event(event, hint=hint)
reraise(*exc_info)
finally:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@untitaker In this "finally" block, the code to stop the thread basically works in case of :-

  1. A successful run of lambda function.
  2. When an exception is there in the code.
    But, it doesn't stop the thread in case of timeout, which is an expected behavior.

By "stop the thread" I mean explicitly stopping a thread by sending a stop signal.

if timeout_thread:
timeout_thread.stop()

return sentry_handler # type: ignore

Expand Down
3 changes: 3 additions & 0 deletions sentry_sdk/integrations/gcp.py
View file
Open in desktop
Expand Up @@ -63,6 +63,7 @@ def sentry_func(functionhandler, event, *args, **kwargs):
_make_request_event_processor(event, configured_time, initial_time)
)
scope.set_tag("gcp_region", environ.get("FUNCTION_REGION"))
timeout_thread = None
if (
integration.timeout_warning
and configured_time > TIMEOUT_WARNING_BUFFER
Expand Down Expand Up @@ -93,6 +94,8 @@ def sentry_func(functionhandler, event, *args, **kwargs):
hub.capture_event(event, hint=hint)
reraise(*exc_info)
finally:
if timeout_thread:
timeout_thread.stop()
# Flush out the event queue
hub.flush()

Expand Down
28 changes: 19 additions & 9 deletions sentry_sdk/utils.py
View file
Open in desktop
Expand Up @@ -891,21 +891,31 @@ def __init__(self, waiting_time, configured_timeout):
threading.Thread.__init__(self)
self.waiting_time = waiting_time
self.configured_timeout = configured_timeout
self._stop_event = threading.Event()

def stop(self):
# type: () -> None
self._stop_event.set()

def stopped(self):
# type: () -> Any
return self._stop_event.is_set()

def run(self):
# type: () -> None

time.sleep(self.waiting_time)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The thread doesn't need to sleep for the full waiting_time. When the handler returns and we call timeout_thread.stop(), the thread should wake up and terminate.

That's why I suggested using self._stop_event.wait(self.waiting_time).

https://docs.python.org/2.7/library/threading.html#threading.Event.wait

You can either use the return value or wait or .is_set() to decide whether to raise ServerlessTimeoutWarning.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I figured that while I did testing and found it was still waiting for complete waiting time.
I've made final changes accordingly now.


integer_configured_timeout = int(self.configured_timeout)
if not self.stopped():
integer_configured_timeout = int(self.configured_timeout)

# Setting up the exact integer value of configured time(in seconds)
if integer_configured_timeout < self.configured_timeout:
integer_configured_timeout = integer_configured_timeout + 1
# Setting up the exact integer value of configured time(in seconds)
if integer_configured_timeout < self.configured_timeout:
integer_configured_timeout = integer_configured_timeout + 1

# Raising Exception after timeout duration is reached
raise ServerlessTimeoutWarning(
"WARNING : Function is expected to get timed out. Configured timeout duration = {} seconds.".format(
integer_configured_timeout
# Raising Exception after timeout duration is reached
raise ServerlessTimeoutWarning(
"WARNING : Function is expected to get timed out. Configured timeout duration = {} seconds.".format(
integer_configured_timeout
)
)
)