Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a MotherDuck environment for default config and safety net #174

Open
mehd-io opened this issue Jun 2, 2023 · 2 comments
Open

Create a MotherDuck environment for default config and safety net #174

mehd-io opened this issue Jun 2, 2023 · 2 comments

Comments

@mehd-io
Copy link
Contributor

mehd-io commented Jun 2, 2023

As md environment differs slightly (and hopefully less in the future ;)) from duckDB, I was thinking of creating a dedicated environment class that would be loaded when users specify a path starting with md:
I'm thinking of the following :

  • Check for md token is present
  • Disable transaction by default as not supported.
  • Keep a list of supported extensions by md and warn the users if this goes outside these.

WDYT? Happy to take a stab at it 🙂

@jwills
Copy link
Collaborator

jwills commented Jun 2, 2023

This would be great @mehd-io -- please send an impl my way!

Couple of things for you to keep in mind:

  • You also likely want to disable any plugins, fsspec-based filesysems, and Python jobs-- i.e., in your environment, trying to call load_source or submit_python_job should raise an exception.
  • Even better would be a way to indicate in the table materialization that Python models aren't supported in MD so that the failure would happen at compile time instead of runtime.
  • It would also be nice to have first-class support for some MD constructs in non-MD environments (e.g. local or Buena Vista)-- i.e., being able to cleanly attach an MD share, or upload a local DuckDB file to MD at the end of a dbt-duckdb run.

@jwills
Copy link
Collaborator

jwills commented Jun 2, 2023

Oh and please add an integration test profile for your MD environment so we can run it as part of CI

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants