otel: simple OTEL collector/Prometheus stack for testing purposes

[krissetto]

- What I did
Added a compose stack to be used for testing out OTEL functionality, mostly based on our current documentation PR found here.

- How I did it

• Added compose stack;
• Added basic OTEL collector and prometheus configs;
• Added basic usage doc

- How to verify it
Read the doc and try it out :)

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

[Benehiko]

I'm not sure about this change since it introduces more complexity with external shell scripts. I'd prefer to keep everything inside Go.

Could we not use testcontainers for this instead? :)

[thaJeztah]

Slightly wondering if we need the scripts at all, as they (effectively) are just a docker compose up?

(I know we have a bunch of ahack/ script already though, but mostly looking if for this one we need more)

[laurazard]

IMO a big part of why I want this change is to make local development easier, so Testcontainers don't really achieve that. I'm okay with shell scripts are they're portable and easy to work with and don't require bringing in additional dependencies, but if atm they are really just doing a compose up, compose down, etc., then I'm okay with holding off on them and just adding the Compose file, and thinking about the rest later if we feel the need for them.

[thaJeztah]

Yes, agreed on the "local development" part. It's usually also my reservation on things like mage as replacement for make; it's cool as it's in Go (and can be flexible), but also means that go is now a requirement on your local machine (and where possible, we should try to still allow for a dockerized workflow, so that there's no real requirements other than "have docker installed")

[laurazard]

LGTM, with a suggestion to remove the shell scripts.

[krissetto]

I'm not super fond of the scripts either, I was trying to follow a bit of what already existed while keeping things as simple s possible for the community (eg. i think buildx has a script which completely wraps compose, for example, and i thought that was a bit too much).
If we agree on removing them completely, I'll adjust the usage.md to reflect that.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 61.05%. Comparing base (b982833) to head (c91c271).
Report is 28 commits behind head on master.

❗ Current head c91c271 differs from pull request most recent head e1dcc19. Consider uploading reports for the commit e1dcc19 to get more accurate results

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #5026   +/-   ##
=======================================
  Coverage   61.05%   61.05%           
=======================================
  Files         295      295           
  Lines       20647    20647           
=======================================
  Hits        12605    12605           
  Misses       7146     7146           
  Partials      896      896           

[Benehiko]

LGTM

[krissetto]

ps i added aspire dashboard here as well so we can test both and see what we want to stick with

[thaJeztah]

Not a blocker for this PR, but I know that bind-mounting files can sometimes be problematic, so looking if we could have a directory to mount (which contains the config-file). Looks like /etc/prometheus already has some files in the image though, so if we do, we probably should use a different location for the config-file (but good news is that we already define it as a custom location above, so ... could still be an option);

docker run --rm --entrypoint /bin/sh prom/prometheus:latest -c 'ls -l /etc/prometheus'
total 4
lrwxrwxrwx    1 nobody   nobody          39 Apr 10 14:30 console_libraries -> /usr/share/prometheus/console_libraries
lrwxrwxrwx    1 nobody   nobody          31 Apr 10 14:30 consoles -> /usr/share/prometheus/consoles/
-rw-r--r--    1 nobody   nobody         934 Apr 10 14:21 prometheus.yml

The alternative could be to define it as a config: in the compose-file (but those are still implemented as bind-mounts in compose, so won't solve the "don't bind individual files" part)

[thaJeztah]

Same for this one (also not a blocker)