Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement aws.ecs.* resource attributes #2626

Merged
merged 11 commits into from Nov 23, 2022

Conversation

mmanciop
Copy link
Contributor

@mmanciop mmanciop commented Aug 6, 2022

Implemented retrieving the aws.ecs.* resource attributes in go.opentelemetry.io/detectors/aws/ecs based on the ECS Metadata v4 endpoint.

detectors/aws/ecs/ecs_test.go Outdated Show resolved Hide resolved
detectors/aws/ecs/ecs_test.go Outdated Show resolved Hide resolved
@codecov
Copy link

codecov bot commented Aug 7, 2022

Codecov Report

Merging #2626 (f3c4f53) into main (15ceaa2) will decrease coverage by 0.7%.
The diff coverage is 0.0%.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##            main   #2626     +/-   ##
=======================================
- Coverage   69.7%   68.9%   -0.8%     
=======================================
  Files        147     147             
  Lines       6806    6875     +69     
=======================================
- Hits        4746    4743      -3     
- Misses      1944    2014     +70     
- Partials     116     118      +2     
Impacted Files Coverage Δ
detectors/aws/ecs/ecs.go 18.0% <0.0%> (-36.8%) ⬇️

@mmanciop
Copy link
Contributor Author

mmanciop commented Aug 12, 2022

There's discussion on open-telemetry/opentelemetry-java#4574 (comment) whether the ARNs like the one this PR builds are actually correct. Until we are use, let's not merge :D

@mmanciop mmanciop marked this pull request as draft August 12, 2022 15:30
@mmanciop mmanciop marked this pull request as ready for review August 12, 2022 16:00
@mmanciop
Copy link
Contributor Author

Update: The uncertainty about ARNs seems sorted out

@bryan-aguilar
Copy link
Contributor

bryan-aguilar commented Aug 15, 2022

Can we add test cases that utilize the fargate examples responses also?
Edit: If these are the same we can disregard. I am still digging into the differences if any, and whether or not these additional test cases would help.

@bryan-aguilar
Copy link
Contributor

Looks like there may be slight differences between fargate and ec2 metadata response. As a sanity check can we add those example responses linked in the comment above to the unit tests?

@mmanciop
Copy link
Contributor Author

Looks like there may be slight differences between fargate and ec2 metadata response. As a sanity check can we add those example responses linked in the comment above to the unit tests?

Done in 2ef2026. There are no factual differences that this patch cares about.

Copy link
Contributor

@bryan-aguilar bryan-aguilar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the ADOT team perspective this looks good to go.
Any long term concerns with using AWS ECS Metadata Go as it describes itself being built for fargate services?

CHANGELOG.md Show resolved Hide resolved
Comment on lines +175 to +177
semconv.AWSLogGroupNamesKey.String(logsOptions.AwsLogsGroup),
semconv.AWSLogGroupARNsKey.String(fmt.Sprintf("arn:aws:logs:%s:%s:log-group:%s:*", logsRegion, awsAccount, logsOptions.AwsLogsGroup)),
semconv.AWSLogStreamNamesKey.String(logsOptions.AwsLogsStream),
semconv.AWSLogStreamARNsKey.String(fmt.Sprintf("arn:aws:logs:%s:%s:log-group:%s:log-stream:%s", logsRegion, awsAccount, logsOptions.AwsLogsGroup, logsOptions.AwsLogsStream)),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
semconv.AWSLogGroupNamesKey.String(logsOptions.AwsLogsGroup),
semconv.AWSLogGroupARNsKey.String(fmt.Sprintf("arn:aws:logs:%s:%s:log-group:%s:*", logsRegion, awsAccount, logsOptions.AwsLogsGroup)),
semconv.AWSLogStreamNamesKey.String(logsOptions.AwsLogsStream),
semconv.AWSLogStreamARNsKey.String(fmt.Sprintf("arn:aws:logs:%s:%s:log-group:%s:log-stream:%s", logsRegion, awsAccount, logsOptions.AwsLogsGroup, logsOptions.AwsLogsStream)),
semconv.AWSLogGroupARNsKey.StringSlice([]string{fmt.Sprintf("arn:aws:logs:%s:%s:log-group:%s:*", logsRegion, awsAccount, logsOptions.AwsLogsGroup)}),
semconv.AWSLogStreamARNsKey.StringSlice([]string{fmt.Sprintf("arn:aws:logs:%s:%s:log-group:%s:log-stream:%s", logsRegion, awsAccount, logsOptions.AwsLogsGroup, logsOptions.AwsLogsStream)}),

These are duplicative, so I'd suggest only including the ARN as the name can be extracted from it.

The values are expected to be string[].

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

String slices introduced in 4e857dd. I am not sold on skipping attributes though, see #2626 (comment).

detectors/aws/ecs/ecs.go Show resolved Hide resolved
detectors/aws/ecs/ecs.go Show resolved Hide resolved
detectors/aws/ecs/ecs.go Show resolved Hide resolved
detectors/aws/ecs/ecs_test.go Outdated Show resolved Hide resolved
@mmanciop
Copy link
Contributor Author

@Aneurysm9 about the duplication of Log Group|Stream Name|ARN attributes, I implemented the spec: https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/cloud_provider/aws/logs/ . They may be duplicative (provided that the backend behind does processing, which is not the case with some OSS backends that have only limited query capabilities), but at the end of the day, if we wanna drop them, I'd rather first they get dropped from the spec.

@Aneurysm9
Copy link
Member

All of the log group/stream fields are optional. There is no requirement to include all of them if any of them are included.

@mmanciop
Copy link
Contributor Author

All of the log group/stream fields are optional. There is no requirement to include all of them if any of them are included.

IMO, this would wreak havoc with less sophisticated tracing backends that do not have processing capabilities to normalize the attributes, as the same Log stream or group could be represented with different values.

@mmanciop mmanciop force-pushed the aws_ecs_resource_attributes branch 2 times, most recently from a47260b to f379b94 Compare September 17, 2022 05:45
@mmanciop
Copy link
Contributor Author

Rebased on main, solved a bunch of merge conflicts; @Aneurysm9 care to have a look?

@mmanciop mmanciop force-pushed the aws_ecs_resource_attributes branch 2 times, most recently from 2fe086b to 0654cc1 Compare September 18, 2022 07:05
@mmanciop mmanciop changed the title Implemented aws.ecs.* resource attributes Implement aws.ecs.* resource attributes Sep 29, 2022
@Aneurysm9
Copy link
Member

@mmanciop This requires an additional approval from @open-telemetry/go-approvers before it can be merged.

"strings"

ecsmetadata "github.com/brunoscheufler/aws-ecs-metadata-go"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This repository has 3 stars and 14 commits. I wonder how reliable it can be in the future.
It's also fairly simple. So we can also extract its content and bring it back into this package later on if necessary.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was committing fixed to that repo. The maintainer was nice and responsive. If worse comes to worse, can always be incorporated here.

@mmanciop mmanciop requested review from bryan-aguilar and removed request for hanyuancheung October 13, 2022 07:48
@mmanciop
Copy link
Contributor Author

mmanciop commented Oct 13, 2022

@Aneurysm9 does this ring a bell?

panic: httptest: failed to listen on a port: listen tcp6 [::1]:0: socket: The requested service provider could not be loaded or initialized. [recovered]
panic: httptest: failed to listen on a port: listen tcp6 [::1]:0: socket: The requested service provider could not be loaded or initialized.

I copied the test code from the Ec2Detector and the tests pass on my Mac; am I perhaps missing something in the CI configs?

(The fact that the CI does not trigger every time I push a commit makes it is even more daunting to troubleshoot...)

Michele Mancioppi added 2 commits October 23, 2022 14:49
After a surreal session of debugging, it turns out that httptest.NewServer fails on Windows by failing to bind the server socket unless the GoLang package name contains "test". To deal with that, I split the tests between "integration tests" needing a HTTP server into "ecs/test", and the other in "ecs".

In order to work around the need of being resilient to the lack of /proc/self/cgroup (which happens in our tests but, above all, when running containers on Windows), the ECVS detector is now more lenient to not finding the container id.
@mmanciop mmanciop requested review from hanyuancheung and bryan-aguilar and removed request for bryan-aguilar and hanyuancheung October 24, 2022 15:33
@Aneurysm9 Aneurysm9 merged commit 01b39e7 into open-telemetry:main Nov 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants