[docs] - Improve Pipes API docs

[erinkcochran87]

Summary & Motivation

This PR adds information to the Pipes and dagster-pipes API references to clarify where and how each set of APIs should be used. Previously, the introductions for these references were vague and confusing - it was difficult to tell which set of APIs you needed and when.

I've also added broken the APIs up into sections, grouped by type, and added information about what each group does. I've also added links to appropriate guides to make cross-referencing content easier.

After:

Preview links: Dagster Pipes reference; dagster-pipes reference

Before:

How I Tested These Changes

👀 , local

[github-actions]

Deploy preview for dagster-docs ready!

Preview available at https://dagster-docs-2t3wjignt-elementl.vercel.app
https://erin-pipes-api-experiment.dagster.dagster-docs.io

Direct link to changed pages:

• https://dagster-docs-2t3wjignt-elementl.vercel.app
  https://erin-pipes-api-experiment.dagster.dagster-docs.io/concepts/automation/schedules/testing

[schrockn]

Important to note that:

1. Writing and using pipes clients
2. Writing code in the external process that just streams metadata back to Dagster.

They are two different use cases targeted to different personas. The "platform owner" more likely to be working on and setting up clients; the "pipeline builder" will be including dagster-pipes in the external environment. Critical point: when someone includes dagster-pipes it does not include the dagster library.

So while the previous organization wasn't ideal merging all the pipes client stuff (which is much more complex) with dagster-pipes API reference also not ideal.

[schrockn]

Also heads up that I'm fixing pipes docs issues: #21719

[erinkcochran87]

@schrockn Heard. Took another swing at this - basically left all the formatting, but moved things back to their separate pages and beefed up the intros. I called out exactly where the APIs in the reference should be used and added callouts for users who may be looking for the other reference.

For example, I added this to the page for dagster-pipes:

...
Looking to set up a Pipes client in Dagster? Refer to the Dagster Pipes API reference.

Note: This library isn't included with dagster and must be installed separately.

Updated dagster-pipes reference:

Updated Dagster Pipes reference:

[schrockn]

Looking much better! My final recommendation is that we further subdivide docs into 1) stuff that people need to know if they are consuming an integration versus 2) people building a new integration (much more advanced)

[schrockn]

I think this can also mention that pipes passes parameters to the external process from the orchestration environment and you can access them there. It's a key value prop of the system.

Re-requesting to get back in queue

[erinkcochran87]

@schrockn Made the changes you asked for! Let me know if you have specific wording in mind and I'll make the updates.

[schrockn]

Looks great! Please heed final comment. 🙏🏻