Lambda layer crashes when LambdaInsights is enabled. #6293

kitsunde · 2022-11-24T15:47:10Z

Is there an existing issue for this?

I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
I have reviewed the documentation https://docs.sentry.io/
I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?

Sentry Saas (sentry.io)

Which package are you using?

@sentry/serverless

SDK Version

arn:aws:lambda:ap-southeast-1:943013980633:layer:SentryNodeServerlessSDK:92

Framework Version

node 14.x

Link to Sentry event

No response

Steps to Reproduce

Since #6093 was merged 22 days ago there has been new releases, but the issue seem to persists.

Given:

Layers: 
  - arn:aws:lambda:ap-southeast-1:943013980633:layer:SentryNodeServerlessSDK:92
  - arn:aws:lambda:ap-southeast-1:580247275435:layer:LambdaInsightsExtension:21

The lambda crashes with:

+ exec /opt/relay/relay run --mode=proxy --shutdown-timeout=2 --aws-runtime-api=127.0.0.1:9001
LOGS	Name: cloudwatch_lambda_agent	State: Subscribed	Types: [platform]
2022-11-24T15:42:47Z [relay::setup] INFO: launching relay without config folder
2022-11-24T15:42:47Z [relay::setup] INFO:   relay mode: proxy
2022-11-24T15:42:47Z [relay::setup] INFO:   relay id: -
2022-11-24T15:42:47Z [relay::setup] INFO:   public key: -
2022-11-24T15:42:47Z [relay::setup] INFO:   log level: INFO
2022-11-24T15:42:47Z [relay_server::actors::outcome] INFO: Configured to emit outcomes as client reports
2022-11-24T15:42:47Z [relay_server::actors::upstream] INFO: upstream relay started
2022-11-24T15:42:47Z [relay_server::actors::outcome] INFO: OutcomeProducer started.
2022-11-24T15:42:47Z [relay_server::actors::processor] INFO: starting 5 envelope processing workers
2022-11-24T15:42:47Z [relay_server::actors::envelopes] INFO: envelope manager started
2022-11-24T15:42:47Z [relay_server::actors::project_cache] INFO: project cache started
2022-11-24T15:42:47Z [relay_server::actors::relays] INFO: key cache started
2022-11-24T15:42:47Z [relay_aws_extension::aws_extension] INFO: AWS extension started
2022-11-24T15:42:47Z [relay_aws_extension::aws_extension] INFO: Registering AWS extension on http://127.0.0.1:9001/2020-01-01/extension
2022-11-24T15:42:47Z [relay_log::utils] ERROR: bind to interface failed
caused by: Address already in use (os error 98)
EXTENSION	Name: cloudwatch_lambda_agent	State: Ready	Events: [INVOKE,SHUTDOWN]
EXTENSION	Name: sentry-lambda-extension	State: Started	Events: []
+ exec /opt/relay/relay run --mode=proxy --shutdown-timeout=2 --aws-runtime-api=127.0.0.1:9001
LOGS	Name: cloudwatch_lambda_agent	State: Already subscribed	Types: [platform]
2022-11-24T15:42:48Z [relay::setup] INFO: launching relay without config folder
2022-11-24T15:42:48Z [relay::setup] INFO:   relay mode: proxy
2022-11-24T15:42:48Z [relay::setup] INFO:   relay id: -
2022-11-24T15:42:48Z [relay::setup] INFO:   public key: -
2022-11-24T15:42:48Z [relay::setup] INFO:   log level: INFO
2022-11-24T15:42:48Z [relay_server::actors::outcome] INFO: Configured to emit outcomes as client reports
2022-11-24T15:42:48Z [relay_server::actors::upstream] INFO: upstream relay started
2022-11-24T15:42:48Z [relay_server::actors::processor] INFO: starting 5 envelope processing workers
2022-11-24T15:42:48Z [relay_server::actors::outcome] INFO: OutcomeProducer started.
2022-11-24T15:42:48Z [relay_server::actors::envelopes] INFO: envelope manager started
2022-11-24T15:42:48Z [relay_server::actors::project_cache] INFO: project cache started
2022-11-24T15:42:48Z [relay_server::actors::relays] INFO: key cache started
2022-11-24T15:42:48Z [relay_aws_extension::aws_extension] INFO: AWS extension started
2022-11-24T15:42:48Z [relay_aws_extension::aws_extension] INFO: Registering AWS extension on http://127.0.0.1:9001/2020-01-01/extension
2022-11-24T15:42:48Z [relay_log::utils] ERROR: bind to interface failed
caused by: Address already in use (os error 98)
EXTENSION	Name: cloudwatch_lambda_agent	State: Ready	Events: [INVOKE,SHUTDOWN]
EXTENSION	Name: sentry-lambda-extension	State: Started	Events: []
START RequestId: 9ef8dc2d-cadf-4d9b-ac67-2a4febd4a485 Version: $LATEST
RequestId: 9ef8dc2d-cadf-4d9b-ac67-2a4febd4a485 Error: exit status 1
Extension.Crash
END RequestId: 9ef8dc2d-cadf-4d9b-ac67-2a4febd4a485
REPORT RequestId: 9ef8dc2d-cadf-4d9b-ac67-2a4febd4a485	Duration: 747.05 ms	Billed Duration: 748 ms	Memory Size: 8192 MB	Max Memory Used: 46 MB	
XRAY TraceId: 1-637f90f7-5ca344660b289c57346e416c	SegmentId: 640f190e2b15ad2f	Sampled: true

If I remove layer:LambdaInsightsExtension the lambda now runs. The last known version that works in this configuration is sentry layer 68.

Expected Result

Not to crash

Actual Result

boom

The text was updated successfully, but these errors were encountered:

AbhiPrasad · 2022-11-24T20:01:21Z

It seems that the LambdaInsightsExtension is using the same port - @antonpirker could you take a look?

kitsunde · 2022-11-24T20:06:18Z

To be clear, this is the same issue as mentioned before the port change in the lambda layer: #5752 (comment)

github-actions · 2022-12-16T00:22:54Z

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

Lms24 · 2023-01-04T10:42:08Z

@sl0thentr0py Since Anton is still on vacation, would you mind taking a quick look at this as I think you already did once in #6093?

sl0thentr0py · 2023-01-04T10:45:14Z

the port is already changed, so this should be fine with latest layer

kitsunde · 2023-01-04T11:02:21Z

the port is already changed, so this should be fine with latest layer

I made the change 22 days after your release as mentioned above, at the time of writing this is using the latest layer following up on a different reported issue after the change went in.

Can you actually confirm it was in by the time that lambda layer was released, and that it works with lambda insights?

If you tell me sentry's layer runs fine with lambda insights, I will recheck this and reproduce it in a project for you.

kitsunde · 2023-01-04T11:21:48Z

@Lms24 Can this remain open, I've rechecked it.

Same issue on arn:aws:lambda:ap-southeast-1:943013980633:layer:SentryNodeServerlessSDK:102 (latest) with arn:aws:lambda:ap-southeast-1:580247275435:layer:LambdaInsightsExtension:21 (latest)

Test Event Name
test

Response
{
  "errorType": "Extension.Crash",
  "errorMessage": "RequestId: bee562c2-2bf3-4340-b33e-28323d6e867d Error: exit status 1"
}

Function Logs
+ exec /opt/relay/relay run --mode=proxy --shutdown-timeout=2 --aws-runtime-api=127.0.0.1:9001
LOGS	Name: cloudwatch_lambda_agent	State: Already subscribed	Types: [platform]
2023-01-04T11:17:44Z [relay::setup] INFO: launching relay without config folder
2023-01-04T11:17:44Z [relay::setup] INFO:   relay mode: proxy
2023-01-04T11:17:44Z [relay::setup] INFO:   relay id: -
2023-01-04T11:17:44Z [relay::setup] INFO:   public key: -
2023-01-04T11:17:44Z [relay::setup] INFO:   log level: INFO
2023-01-04T11:17:45Z [relay_server::actors::outcome] INFO: Configured to emit outcomes as client reports
2023-01-04T11:17:45Z [relay_server::actors::processor] INFO: starting 2 envelope processing workers
2023-01-04T11:17:45Z [relay_server::actors::upstream] INFO: upstream relay started
2023-01-04T11:17:45Z [relay_server::actors::envelopes] INFO: envelope manager started
2023-01-04T11:17:46Z [relay_server::actors::outcome] INFO: OutcomeProducer started.
2023-01-04T11:17:46Z [relay_server::actors::outcome_aggregator] INFO: outcome aggregator started
2023-01-04T11:17:46Z [relay_server::actors::project_cache] INFO: project cache started
2023-01-04T11:17:46Z [relay_server::actors::relays] INFO: key cache started
2023-01-04T11:17:46Z [relay_aws_extension::aws_extension] INFO: AWS extension started
2023-01-04T11:17:46Z [relay_aws_extension::aws_extension] INFO: Registering AWS extension on http://127.0.0.1:9001/2020-01-01/extension
2023-01-04T11:17:46Z [relay_metrics::aggregation] INFO: aggregator started
2023-01-04T11:17:46Z [relay_log::utils] ERROR: bind to interface failed
caused by: Address already in use (os error 98)
EXTENSION	Name: cloudwatch_lambda_agent	State: Ready	Events: [INVOKE,SHUTDOWN]
EXTENSION	Name: sentry-lambda-extension	State: Started	Events: []
START RequestId: bee562c2-2bf3-4340-b33e-28323d6e867d Version: $LATEST
RequestId: bee562c2-2bf3-4340-b33e-28323d6e867d Error: exit status 1
Extension.Crash
END RequestId: bee562c2-2bf3-4340-b33e-28323d6e867d
REPORT RequestId: bee562c2-2bf3-4340-b33e-28323d6e867d	Duration: 4420.22 ms	Billed Duration: 4421 ms	Memory Size: 128 MB	Max Memory Used: 45 MB	
XRAY TraceId: 1-63b56056-7d5745b03fd2ac750374b913	SegmentId: 00f538954cd146da	Sampled: true

Request ID
bee562c2-2bf3-4340-b33e-28323d6e867d

The code is just:

const Sentry = require('@sentry/serverless');

Sentry.AWSLambda.init();

async function main(event) {
  return { statusCode: 200, body: 'pog' };
}

exports.handler = Sentry.AWSLambda.wrapHandler(main, {
  ignoreSentryErrors: true,
});

🤷‍♂️

kitsunde · 2023-01-04T12:11:25Z

Okay I've found the issue. @sl0thentr0py

The layer isn't being built with https://github.com/getsentry/action-build-aws-lambda-extension/blob/68818ccadd075f74010e80fe350f205554a16d99/action.yml#L41-L57

You can confirm it by fetching the layer from AWS:

curl $(aws lambda get-layer-version-by-arn --arn arn:aws:lambda:ap-southeast-1:943013980633:layer:SentryNodeServerlessSDK:102 --query Content.Location --output text) -o lambda-layer.zip
unzip lambda-layer.zip
cat extensions/sentry-lambda-extension

Outputs:

#!/bin/bash
set -xeuo pipefail
exec /opt/relay/relay run --mode=proxy --shutdown-timeout=2 --aws-runtime-api="$AWS_LAMBDA_RUNTIME_API"

Seems like it's missing both --upstream-dsn (whatever that does) and --port.

sl0thentr0py · 2023-01-04T12:21:35Z

yea just found that as well, apologies and thanks for investigating.
will fix soon.

sl0thentr0py · 2023-01-04T12:25:46Z

ok something was lost in communication with the action author, sorry for that.
I bumped the tags on https://github.com/getsentry/action-build-aws-lambda-extension/tags now, so the next release will for reals pick up the changes.

kitsunde · 2023-01-04T12:30:06Z

Cool cool, thanks. :)

sl0thentr0py · 2023-01-04T12:34:48Z

good to go, I think they're gonna release today so good timing :)
https://github.com/getsentry/sentry-javascript/actions/runs/3837813629/jobs/6533975273#step:8:2867

kitsunde added the Type: Bug label Nov 24, 2022

getsantry bot added the Status: Untriaged label Nov 24, 2022

AbhiPrasad added Status: Needs Reproduction Feature: Serverless labels Nov 24, 2022

getsantry bot removed the Status: Untriaged label Nov 24, 2022

github-actions bot added the Status: Stale label Dec 16, 2022

github-actions bot closed this as completed Dec 24, 2022

Lms24 reopened this Jan 4, 2023

Lms24 closed this as completed Jan 4, 2023

sl0thentr0py reopened this Jan 4, 2023

github-actions bot removed the Status: Stale label Jan 5, 2023

smeubank closed this as completed Jan 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lambda layer crashes when LambdaInsights is enabled. #6293

Lambda layer crashes when LambdaInsights is enabled. #6293

kitsunde commented Nov 24, 2022

AbhiPrasad commented Nov 24, 2022

kitsunde commented Nov 24, 2022

github-actions bot commented Dec 16, 2022

Lms24 commented Jan 4, 2023

sl0thentr0py commented Jan 4, 2023

kitsunde commented Jan 4, 2023

kitsunde commented Jan 4, 2023

kitsunde commented Jan 4, 2023 •

edited

sl0thentr0py commented Jan 4, 2023

sl0thentr0py commented Jan 4, 2023

kitsunde commented Jan 4, 2023

sl0thentr0py commented Jan 4, 2023

Lambda layer crashes when LambdaInsights is enabled. #6293

Lambda layer crashes when LambdaInsights is enabled. #6293

Comments

kitsunde commented Nov 24, 2022

Is there an existing issue for this?

How do you use Sentry?

Which package are you using?

SDK Version

Framework Version

Link to Sentry event

Steps to Reproduce

Expected Result

Actual Result

AbhiPrasad commented Nov 24, 2022

kitsunde commented Nov 24, 2022

github-actions bot commented Dec 16, 2022

Lms24 commented Jan 4, 2023

sl0thentr0py commented Jan 4, 2023

kitsunde commented Jan 4, 2023

kitsunde commented Jan 4, 2023

kitsunde commented Jan 4, 2023 • edited

sl0thentr0py commented Jan 4, 2023

sl0thentr0py commented Jan 4, 2023

kitsunde commented Jan 4, 2023

sl0thentr0py commented Jan 4, 2023

kitsunde commented Jan 4, 2023 •

edited