mdatagen: add wildcard name matching for configs #10065

braydonk · 2024-05-01T19:26:23Z

Description

This PR adds support for wildcard name matching.

Generated MetricsConfig and ResourceAttributesConfig will now support providing names not just as full names, but also using * wildcards and multimatching with {x,y,etc}. This allows you to apply configs to groups of metrics and resource attributes, simplifying configs.

Link to tracking issue

Fixes #10074

Testing

I go installed mdatagen from this PR and ran it on a receiver in opentelemetry-collector-contrib and was able to use wildcard matching to enable/disable groups of metrics as expected.

Documentation

I have written a specification in a gist: https://gist.github.com/braydonk/ccb6775331fdd5dca91a650330b9839f

I am not sure where this documentation should live.

codecov · 2024-05-01T20:15:53Z

Codecov Report

Attention: Patch coverage is 33.33333% with 8 lines in your changes are missing coverage. Please review.

Project coverage is 92.43%. Comparing base (c7c3401) to head (d82df38).
Report is 3 commits behind head on main.

❗ Current head d82df38 differs from pull request most recent head 08cf72d

Please upload reports for the commit 08cf72d to get more accurate results.

Files	Patch %	Lines
cmd/mdatagen/main.go	33.33%	4 Missing and 4 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #10065      +/-   ##
==========================================
- Coverage   92.59%   92.43%   -0.16%     
==========================================
  Files         387      387              
  Lines       18198    18264      +66     
==========================================
+ Hits        16850    16883      +33     
- Misses       1007     1031      +24     
- Partials      341      350       +9

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

braydonk · 2024-05-01T20:19:55Z

I think that code coverage change is fine here because none of the return err cases for template file generation are covered anyway.

braydonk · 2024-05-02T12:20:23Z

The changelog issue can only be resolved once the issue is moved from contrib to here.

dmitryax · 2024-05-04T16:29:47Z

cmd/mdatagen/templates/match.go.tmpl

+	"strings"
+)
+
+var ErrNotMultimatch = errors.New("this index doesn't represent a valid multimatch pattern")


I don't see a reason to export any API in this file. We should only export config field types and Unmarshal methods

None of the API is exported in the rewrite 3f0a4e8

dmitryax · 2024-05-04T16:38:32Z

cmd/mdatagen/templates/match.go.tmpl

+		for _, name := range matchNames {
+			if pattern.Match(name) {
+				matched = append(matched, name)
+			}


We should fail unmarshalling if we run into a pattern that doesn't match any metric names. It'll be aligned with the strict matching behavior

I will add this in.

Done in d82df38

dmitryax · 2024-05-04T16:41:36Z

cmd/mdatagen/templates/match.go.tmpl

+				patternScan.idx += 2
+			}
+
+		case '{':


Can you please point to some specifications for this sort of pattern matching? Does it usually go with wildcard matching?

Added in the description of the PR, but link here: https://gist.github.com/braydonk/ccb6775331fdd5dca91a650330b9839f

github-actions · 2024-05-19T03:16:01Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

dmitryax · 2024-05-21T19:16:01Z

@braydonk, are you still working on this? Just checking. Let me know if you have any concerns with my comments

braydonk · 2024-05-21T19:40:13Z

Hi @dmitryax yes I am. I realized that I forgot to mention that I am writing a full spec to explain how it works, and while I was doing that found a number of edge cases that prompted me to rewrite the parser. Taking longer than expected. I am still working on this though and will keep it unstale.

This PR adds support for wildcard name matching. Generated MetricsConfig and ResourceAttributesConfig will now support providing names not just as full names, but also using `*` wildcards and multimatching with `{x,y,etc}`. This allows you to apply configs to groups of metrics and resource attributes, simplifying configs. Signed-off-by: braydonk <braydonk@google.com>

braydonk · 2024-05-28T16:17:54Z

What the actual errors look like from collector output is not super obvious from the PR, so here are some examples. I used the hostmetrics process scraper in my example.

Not matching any names with the patterns

hostmetrics:
  collection_interval: 10s
  scrapers:
    process:
      metrics:
        "nothing.*":
          enabled: false

$ ./otelcontribcol --config config.yaml 
Error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:

* error decoding 'receivers': error reading configuration for "hostmetrics": error reading settings for scraper type "process": 1 error(s) decoding:

* error decoding 'metrics': specified patterns don't match any names
2024/05/28 16:06:44 collector server run finished with error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:

* error decoding 'receivers': error reading configuration for "hostmetrics": error reading settings for scraper type "process": 1 error(s) decoding:

* error decoding 'metrics': specified patterns don't match any names

Parsing errors in a pattern

hostmetrics:
  collection_interval: 10s
  scrapers:
    process:
      metrics:
        "a_.*":
          enabled: false

$ ./otelcontribcol --config config.yaml
Error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:

* error decoding 'receivers': error reading configuration for "hostmetrics": error reading settings for scraper type "process": 1 error(s) decoding:

* error decoding 'metrics': pattern parsing error:
a_.*
  ^
invalid pattern: unexpected token
2024/05/28 16:06:52 collector server run finished with error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:

* error decoding 'receivers': error reading configuration for "hostmetrics": error reading settings for scraper type "process": 1 error(s) decoding:

* error decoding 'metrics': pattern parsing error:
a_.*
  ^
invalid pattern: unexpected token

braydonk · 2024-05-28T16:38:25Z

I'm gonna wait to address the conflicts, cause this is just going to keep coming up continuously throughout the review.

braydonk requested a review from a team as a code owner May 1, 2024 19:26

braydonk requested a review from Aneurysm9 May 1, 2024 19:26

braydonk force-pushed the wildcard_names branch 2 times, most recently from 026ad18 to 2ed7ec4 Compare May 1, 2024 20:07

braydonk mentioned this pull request May 1, 2024

[cmd/mdatagen] Ability to disable/enable all metrics #10074

Open

dmitryax reviewed May 4, 2024

View reviewed changes

github-actions bot added the Stale label May 19, 2024

dmitryax removed the Stale label May 21, 2024

braydonk added 4 commits May 28, 2024 15:59

fix issue number

5c9fb7e

matching code rewritten

19f3d88

fail parsing if there are no matches

d82df38

braydonk force-pushed the wildcard_names branch from 3f0a4e8 to d82df38 Compare May 28, 2024 16:02

rerun gogenerate

08cf72d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mdatagen: add wildcard name matching for configs #10065

mdatagen: add wildcard name matching for configs #10065

braydonk commented May 1, 2024 •

edited

codecov bot commented May 1, 2024 •

edited

braydonk commented May 1, 2024

braydonk commented May 2, 2024

dmitryax May 4, 2024

braydonk May 28, 2024

dmitryax May 4, 2024

braydonk May 28, 2024

braydonk May 28, 2024

dmitryax May 4, 2024

braydonk May 28, 2024

github-actions bot commented May 19, 2024

dmitryax commented May 21, 2024

braydonk commented May 21, 2024

braydonk commented May 28, 2024 •

edited

braydonk commented May 28, 2024

mdatagen: add wildcard name matching for configs #10065

Are you sure you want to change the base?

mdatagen: add wildcard name matching for configs #10065

Conversation

braydonk commented May 1, 2024 • edited

Description

Link to tracking issue

Testing

Documentation

codecov bot commented May 1, 2024 • edited

Codecov Report

braydonk commented May 1, 2024

braydonk commented May 2, 2024

dmitryax May 4, 2024

Choose a reason for hiding this comment

braydonk May 28, 2024

Choose a reason for hiding this comment

dmitryax May 4, 2024

Choose a reason for hiding this comment

braydonk May 28, 2024

Choose a reason for hiding this comment

braydonk May 28, 2024

Choose a reason for hiding this comment

dmitryax May 4, 2024

Choose a reason for hiding this comment

braydonk May 28, 2024

Choose a reason for hiding this comment

github-actions bot commented May 19, 2024

dmitryax commented May 21, 2024

braydonk commented May 21, 2024

braydonk commented May 28, 2024 • edited

braydonk commented May 28, 2024

braydonk commented May 1, 2024 •

edited

codecov bot commented May 1, 2024 •

edited

braydonk commented May 28, 2024 •

edited