Otel collector receiver (Component?) guide

[hughesjj]

In opentelemetry.io, we have an existing guide for creating a span receiver. The guide is great – to my understanding, it’s a syndicated version of a blog post made a few years ago, and we’ve done some to keep it up to date. While still useful for current and aspiring collector contributors, we haven’t given it much tender loving care over the years, and the tribal knowledge between the collector sig and what's in the document has drifted a bit.

Motivation

Anecdotally and personally, I have a few ideas for improvement that I’ve workedshopped in the community spec before along with some of my cohorts which I’d like to see us implement in the official public docs.
Most importantly, I wish for the guide to be easier to maintain for the collector-contrib SIG members and similar interested parties.

Proposal

We should break up the “creating an X” guides into a more general flow that partitions individual pages of content out into different parts of the execution cycle (requirements gathering, design, implementation, testing, acceptance, maintenance, etc), with “chapters” or subfolders that can dive into how to do so for any particular component.

The “highest level” docs could be seeding context for the various parts of the execution, leading the reader through the process for what concerns they consider. Once they have this in scope, they can dive into a detailed, guided, comprehensive tutorial for how to execute the particular task, given some arbitrary (illustrative?) choices for the aforementioned “problem context”.

Plan of attack

I’d like to do the high level first before diving into implementation, but we can do this in parallel/pipelined.

working proposal (would make these github issues etc)

Phase 1 would be additive only, in the existing collector/building folder

Phase 1: new content

1. Add content around design considerations for making a receiver
2. add content around mdatagen
3. add content around onus/responsibility for
4. add content around making a metrics receiver (possibly including how to make them cross-signal compatible)
5. add content for modern-ish integration testing

Phase 2: new format

1. organize content in collector/building into a hand-held guide with flow and decision points instead of a "flat" hierarchy. Needs consensus of content first.

[svrnm]

@hughesjj I agree with you that the original page needs a major rework since a lot has changed over time. What we need to discuss how much developer documentation should live in the docs and how much should be left to the repositories. In this particular case I personally lean towards having this on the website, since "building your own collector component" is something more and more people who are technically end-users will look for. They might end up in the contrib repository, but I argue that there are many use cases were folks want to have their "own" thing that might not even be OSS.

A thing I'd like to call out is that we need to be clear on how many pages we want to distribute that content and how we are going to structure them.

cc @open-telemetry/collector-approvers

[hughesjj]

What we need to discuss how much developer documentation should live in the docs and how much should be left to the repositories. In this particular case I personally lean towards having this on the website, since "building your own collector component" is something more and more people who are technically end-users will look for.

That's fair. I can tone down any -contrib specific mention, especially now that -contrib has upstreamed mdatagen. IMO there's still a decent number of "helper" functions/tooling currently exclusive to -contrib which could be worth pointing to, but fair point that we should make nothing in the documentation depend on -contrib patterns or tooling.

A thing I'd like to call out is that we need to be clear on how many pages we want to distribute that content and how we are going to structure them.

Sounds good, what's the domain of "pages" though? I'd assume just the opentelemetry.io website/repo?

If so, IMO, keep this proposed new content in both a collector/building/ and collector/building/reciever folder and link to them from elsewhere if there's ever a relevant reason to do so reason.

If we're talking about an actual outline, my proposal would be

collector/building

1. Introduction to components & otel "primitives"
2. Overview of mdatagen
3. How to compile and test your component

collector/building/receiver

1. Design of a receiver (requirements gathering and introduction to otel receiver "primitives")
2. Scaffolding of a receiver (mdatagen etc)
3. Implementing a receiver (one page each for below)
   3a. Metrics Receiver
   3b. Trace Receiver
   3c. Logs receiver
   3d. Multi-signal-type receiver

[svrnm]

I like this structure, what do you think is a good starting point? We always prefer to have "incremental" upgrades, so instead of having one BIG pr with all the content, let's find the first pieces to start with and then grow from there.

[hughesjj]

My top two picks:

1. Introduction to components & otel "primitives"
2. Overview of mdatagen

Alternatively, reorganize the layout first

[svrnm]

@hughesjj as discussed during the SIG meeting, tackle those 2 topics and then work through that list incrementally, if you see any need to reorganize content and/or replace content please do so!