Add quickstart example for dbt & Dagster #21240

maximearmstrong · 2024-04-16T18:59:51Z

Summary & Motivation

This PR adds a quickstart example for dbt & Dagster. A user that wants to skip the tutorial can copy and paste the code included in the example and run dagster dev.

This example includes the new experimental class DbtProject. Another PR will add more DbtProject examples in the dbt docs and notes that the feature is still experimental.

How I Tested These Changes

BK
+
local

make mdx-full-format
make next-watch-build

maximearmstrong · 2024-04-16T19:00:04Z

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

Add quickstart example for dbt & Dagster #21240 : 3 dependent PRs (#21568 , #21816 , #21817 ) 👈
Add dbt + Dagster+ guide #21229 : 1 other dependent PR (#21950 )
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @maximearmstrong and the rest of your teammates on Graphite

cmpadden · 2024-05-01T21:01:51Z

docs/content/integrations/dbt/quickstart.mdx

+
+---
+
+## Step 2: Create your code location file


It would be nice if we could model this how we've structured our other quickstart projects:

https://github.com/dagster-io/dagster/tree/master/examples/quickstart_snowflake

That way we can have a user scaffold the project using the template:

pip install dagster dagster project from-example \ --name my-dagster-project \ --example quickstart_dbt

That is what we did for our Dagster University dbt starter project:

https://github.com/dagster-io/dagster/tree/master/examples/project_du_dbt_starter

yeah, I'd like to defer away from having users copy+paste large code snippets. ack that we have a history of it, but we can be better.

though I'm okay with not doing this until we un-experimentalize DbtProject and can merge/deprecate the existing dbt example project we have.

rexledesma

I'm in favor of this change. But two things:

We should model this on our existing dagster-dbt project scaffold entrypoint.
The quickstart should just be a subset of the tutorial.

I'm also imagining two quickstart guides:

A quickstart for a brand new Dagster project
A quickstart to integrate into an existing Dagster project

For (1), if they don't have an existing Dagster project, they can use the outputs of the dagster-dbt project scaffold CLI wholesale.

Essentially, we just need to copy/paste steps 1/2/3 from https://docs.dagster.io/integrations/dbt/using-dbt-with-dagster/load-dbt-models.

For (2), if they already have an existing Dagster project, we can just tell them to copy and paste the relevant bits from the generated scaffold to their existing project.

We should split (1) and (2) into separate PRs.

erinkcochran87 · 2024-05-01T21:17:23Z

Note that I plan to chime in once the direction of this is a bit more solid.

tacastillo

Looking good and I'm excited for this!

Most of my comments are nits, the only one that I'd really like addressed is the one about the What's next?

tacastillo · 2024-05-01T21:10:15Z

docs/content/integrations/dbt/quickstart.mdx

+1. To start Dagster's UI, run the following:
+
+   ```shell
+   dagster dev -f definitions.py


Might want to call out that this is assuming that they followed our directive of calling the file definitions.py, probably up top where it's first defined.

tacastillo · 2024-05-01T21:18:10Z

docs/content/integrations/dbt/quickstart.mdx

+    dbt_assets,
+)
+
+RELATIVE_PATH_TO_YOUR_DBT_PROJECT = "relative/path/to/your/dbt-project"


Hmm, I think our users might benefit from some additional hand-holding here.

How do you feel about either:

instructing them to make this file in/adjacent to the dbt project dir?

or giving a more precise example of what the path could be?

tacastillo · 2024-05-01T21:19:19Z

docs/content/integrations/dbt/quickstart.mdx

+In this example, we created a single file to handle a simple use case and run your dbt project. To learn more about our dbt integrations and how to handle more complex use cases, consider the following options:
+
+- Begin the official dbt scaffold tutorial, like the [dbt & Dagster project](/integrations/dbt/using-dbt-with-dagster).
+- Checkout out the Dagster & dbt course in [Dagster University](https://courses.dagster.io).


Our users might hit some decision paralysis here. Which one should they do in what situation and why?

maximearmstrong · 2024-05-02T17:54:07Z

@rexledesma @tacastillo @cmpadden These are very good points. I think using the scaffold would make sense in the quickstart.

Maybe we could have a quickstart section and another one for the copy-paste code snippets? Or maybe we want to avoid copy+paste code altogether?

The goal of Lopp's internal ticket was to add a section where a user could simply copy+paste code.

@slopp What is your take on this? Would a quick start page with a subset of the tutorial be useful in the context you had in mind? Has this request been made by users?

tacastillo · 2024-05-02T17:57:30Z

Ah if that's the scope of the original ask, then yeah I'm down for splitting concerns to prevent creep.

will wait for the verdict, but otherwise, feel free to hit re-request review

yuhan · 2024-05-02T23:15:54Z

Scaffolding is nice and should be encouraged to use, but it sometimes feels heavy to certain new users. But I think having a single file that's copy+paste-able is good - It's handy for us to share how it looks and it's easier for users without a lot of time/patience to digest. So, maybe a path forward is:

Have a dedicated section in the end to show the finished single definitions.py that users can jump directly to and copy+paste if they want to skip the whole setup and explainers. (something like this section but more linkable)
Add a callout at the beginning before "## Step 1: Setup your environment" to the end full example like "skip to the bottom to see the whole thing"
- something like this:

Other than that, I agree with all the points made about scaffold and two paths of quickstart.

## Summary & Motivation This PR updates the quickstart example for dbt + Dagster - it adds the steps to create a brand new Dagster project using the scaffold to wrap a dbt project. This work is the continuation of #21240 ## How I Tested These Changes BK + local ``` make mdx-full-format make next-watch-build ```

re-requesting

rexledesma

sweet, see nits before merging

rexledesma · 2024-05-16T22:32:21Z

docs/content/integrations/dbt/quickstart.mdx

@@ -0,0 +1,264 @@
+---
+title: "Quickstart: Run your dbt project with Dagster"
+description: Get started swiftly with this simple dbt + Dagster example.


nit: no + since that's might be confused with Dagster+

Suggested change

description: Get started swiftly with this simple dbt + Dagster example.

description: Get started swiftly with this simple dbt & Dagster example.

rexledesma · 2024-05-16T22:33:54Z

docs/content/integrations/dbt/quickstart.mdx

+<Note>
+  <strong>Note</strong>: This guide uses the <code>DbtProject</code> class,
+  which is an experimental feature. Visit the{" "}
+  <a href="/\_apidocs/libraries/dagster-dbt">


We should link to the class directly
https://docs.dagster.io/

Suggested change

<a href="/\_apidocs/libraries/dagster-dbt">

<a href="/\_apidocs/libraries/dagster-dbt#dagster_dbt.DbtProject">

rexledesma · 2024-05-16T22:36:27Z

docs/content/integrations/dbt/quickstart.mdx

+  for more info.
+</Note>
+
+If you just want to see the code of the example, skip to the **Create a single Dagster file** section.


This sentence can probably can be omitted, given that it's the default option in "Load your dbt project"

rexledesma · 2024-05-16T22:36:48Z

docs/content/integrations/dbt/quickstart.mdx

+  info.
+</Note>
+
+Install dbt, Dagster, and the Dagster webserver/UI\*\*. Run the following to install everything using pip:


Suggested change

Install dbt, Dagster, and the Dagster webserver/UI\*\*. Run the following to install everything using pip:

Install dbt, Dagster, and the Dagster webserver\*\*. Run the following to install everything using pip:

## Summary & Motivation This PR adds a quickstart example for dbt & Dagster. A user that wants to skip the tutorial can copy and paste the code included in the example and run `dagster dev`. This example includes the new experimental class `DbtProject`. Another PR will add more DbtProject examples in the dbt docs and notes that the feature is still experimental. ## How I Tested These Changes BK + local ``` make mdx-full-format make next-watch-build ```

maximearmstrong mentioned this pull request Apr 16, 2024

Add dbt + Dagster+ guide #21229

Draft

maximearmstrong force-pushed the maxime/ds-164/add-doc-page-for-dbt-dagster-cloud branch from c40f38a to 962ba18 Compare April 18, 2024 17:42

maximearmstrong force-pushed the maxime/ds-169/add-full-copy-pasteable-example-to-dbt-integrate-page branch from bc5c6c8 to e95f9e3 Compare April 18, 2024 17:42

maximearmstrong force-pushed the maxime/ds-164/add-doc-page-for-dbt-dagster-cloud branch from 962ba18 to 03888c5 Compare April 30, 2024 20:34

maximearmstrong force-pushed the maxime/ds-169/add-full-copy-pasteable-example-to-dbt-integrate-page branch from b146c7f to 86de03d Compare April 30, 2024 20:34

maximearmstrong self-assigned this Apr 30, 2024

maximearmstrong marked this pull request as ready for review April 30, 2024 21:35

maximearmstrong requested review from erinkcochran87, yuhan and slopp April 30, 2024 21:35

graphite-app bot added the area: docs Related to documentation in general label Apr 30, 2024

maximearmstrong force-pushed the maxime/ds-164/add-doc-page-for-dbt-dagster-cloud branch from 03888c5 to 082265a Compare May 1, 2024 20:12

maximearmstrong force-pushed the maxime/ds-169/add-full-copy-pasteable-example-to-dbt-integrate-page branch from 82d4a19 to a9d5138 Compare May 1, 2024 20:12

maximearmstrong mentioned this pull request May 1, 2024

Update dbt docs with DbtProject #21568

Draft

cmpadden reviewed May 1, 2024

View reviewed changes

maximearmstrong force-pushed the maxime/ds-164/add-doc-page-for-dbt-dagster-cloud branch from 082265a to c3d66ba Compare May 1, 2024 21:05

maximearmstrong force-pushed the maxime/ds-169/add-full-copy-pasteable-example-to-dbt-integrate-page branch from a9d5138 to c20e75b Compare May 1, 2024 21:05

rexledesma previously requested changes May 1, 2024

View reviewed changes

tacastillo previously requested changes May 1, 2024

View reviewed changes

maximearmstrong force-pushed the maxime/ds-164/add-doc-page-for-dbt-dagster-cloud branch from c3d66ba to 35923aa Compare May 9, 2024 21:02

maximearmstrong force-pushed the maxime/ds-169/add-full-copy-pasteable-example-to-dbt-integrate-page branch from c20e75b to 1f290e7 Compare May 9, 2024 21:02

maximearmstrong mentioned this pull request May 13, 2024

Add brand new Dagster project example to dbt quickstart #21816

Merged

maximearmstrong force-pushed the maxime/ds-164/add-doc-page-for-dbt-dagster-cloud branch from 35923aa to 662d7d0 Compare May 13, 2024 18:43

maximearmstrong force-pushed the maxime/ds-169/add-full-copy-pasteable-example-to-dbt-integrate-page branch from 54ed678 to 88b03c6 Compare May 13, 2024 18:43

maximearmstrong mentioned this pull request May 13, 2024

Add existing Dagster project example to dbt quickstart #21817

Merged

maximearmstrong and others added 14 commits May 14, 2024 18:14

Fix typo in image path

11e7ad9

Update example

816263e

Update example

430e742

Add DbtProject note in quickstart

71ab249

Update what's next and step 3 in quickstart.mdx

815ecaf

Update relative dbt project path in quickstart.mdx

5cfd85a

Update quickstart.mdx

54f8290

Update UI step

688791c

Update DbtProject note

86a46c2

Fix note

cf69aff

Use underscore in dir names

4f41219

Pull

6213fc4

Update code snippets

f120748

maximearmstrong changed the base branch from maxime/ds-164/add-doc-page-for-dbt-dagster-cloud to master May 16, 2024 20:28

maximearmstrong added 2 commits May 16, 2024 16:31

Rebase on master

e6b837b

Update quickstart

0211d97

maximearmstrong requested a review from rexledesma May 16, 2024 20:58

maximearmstrong added 2 commits May 16, 2024 17:15

Fix deadlink

f35870d

Fix code snippets

9570ef0

maximearmstrong requested review from rexledesma and removed request for rexledesma May 16, 2024 21:53

rexledesma approved these changes May 16, 2024

View reviewed changes

Update quickstart post review

41340b8

maximearmstrong changed the title ~~Add quickstart example for dbt + Dagster~~ Add quickstart example for dbt & Dagster May 17, 2024

maximearmstrong merged commit 0d005f5 into master May 17, 2024
1 of 2 checks passed

maximearmstrong deleted the maxime/ds-169/add-full-copy-pasteable-example-to-dbt-integrate-page branch May 17, 2024 00:14

maximearmstrong mentioned this pull request May 17, 2024

Add cloud action documentation in dbt & Dagster+ guide #21950

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add quickstart example for dbt & Dagster #21240

Add quickstart example for dbt & Dagster #21240

maximearmstrong commented Apr 16, 2024 •

edited

maximearmstrong commented Apr 16, 2024 •

edited

cmpadden May 1, 2024

cmpadden May 1, 2024

tacastillo May 1, 2024

rexledesma left a comment

erinkcochran87 commented May 1, 2024

tacastillo left a comment

tacastillo May 1, 2024

tacastillo May 1, 2024

tacastillo May 1, 2024

maximearmstrong commented May 2, 2024

tacastillo commented May 2, 2024

yuhan commented May 2, 2024

rexledesma left a comment

rexledesma May 16, 2024

rexledesma May 16, 2024

rexledesma May 16, 2024

rexledesma May 16, 2024

	description: Get started swiftly with this simple dbt + Dagster example.
	description: Get started swiftly with this simple dbt & Dagster example.

	<a href="/\_apidocs/libraries/dagster-dbt">
	<a href="/\_apidocs/libraries/dagster-dbt#dagster_dbt.DbtProject">

	Install dbt, Dagster, and the Dagster webserver/UI\\. Run the following to install everything using pip:
	Install dbt, Dagster, and the Dagster webserver\\. Run the following to install everything using pip:

Add quickstart example for dbt & Dagster #21240

Add quickstart example for dbt & Dagster #21240

Conversation

maximearmstrong commented Apr 16, 2024 • edited

Summary & Motivation

How I Tested These Changes

maximearmstrong commented Apr 16, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rexledesma left a comment

Choose a reason for hiding this comment

erinkcochran87 commented May 1, 2024

tacastillo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maximearmstrong commented May 2, 2024

tacastillo commented May 2, 2024

yuhan commented May 2, 2024

rexledesma left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maximearmstrong commented Apr 16, 2024 •

edited

maximearmstrong commented Apr 16, 2024 •

edited