Adding upstream asset as source has correct lineage but fails on run for DBT

[noahtf13]

I have created a very simple DBT model that is downstream of an asset that writes a Pandas df to BigQuery. I want the DBT model to run after that asset is done. The model is as follows (and is just a test):

SELECT 
    *
FROM {{source('dagster', 'upstream')}} 

I've added the upstream asset in models/sources.yml with:

sources:
  - name: dagster
    tables:
      - name: upstream
        meta:
          dagster:
            asset_key: ["upstream"]

And can see the correct lineage when viewing the model in the global asset lineage. Upon running the model that is downstream from the asset that I added to the source, the compiled SQL is looking for what it traditionally does which is the BQ table [source argument #1, source argument #2].

The error is

  404 Not found: Dataset [my_gcp_project]:dagster was not found in location US; reason: notFound, message: Not found: Dataset [my_gcp_project]:dagster was not found in location US

So it is clear that my model is acting as if the source given is a traditional one where I'm just pulling straight from Bigquery, but it also shows the proper lineage in the UI/DAG

Any help would be appreciated I've scrolled through the discussions looking for similar issues as well as following the docs closely but to no avail.

Edit: ...or is that just it. I have to properly define the source (where the data is located in BQ) and link it to the upstream asset. Maybe I need sleep.

[cmpadden]

Hi @noahtf13 - were you able to figure this one out? The message you appended in your Edit sounds the most plausible, in that you must define your big query source that is referenced by the dbt model in {{ source('dagster', 'upstream') }}. Double check your connection parameters to ensure you can read the table.

Other than that, how you've described linking the dbt model to the upstream asset looks good.