-
Notifications
You must be signed in to change notification settings - Fork 69
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Intermediate Relation Fails with External Materialization with Partitions #232
Comments
@jesseginsberg this might seem stupid, but does it work if you do the options as the (lowercase) The logic we use for constructing the external read location is defined here and it's case-sensitive in the config option name it's looking for: https://github.com/jwills/dbt-duckdb/blob/master/dbt/adapters/duckdb/impl.py#L136 |
@jwills well that was an easy solve 😄. Yes, when I use Thanks for the quick response! |
Hi! |
IIRC you can specify any set of key-value pairs that you like in the |
Any boolean options (like |
Thanks @jwills. |
Hello, I'm running into an issue when using the external materialization with the "PARTITION_BY" option.
The dbt run goes ok through the part where it writes out the partitioned files to the file system. But it fails when it reaches the step where it creates the intermediate_relation view within DuckDB. I believe it is the section here: https://github.com/jwills/dbt-duckdb/blob/master/dbt/include/duckdb/macros/materializations/external.sql#L51-L55.
My dbt model called
test_model
looks something like:Looking at the logs, after dbt writes out the partitioned files externally dbt tries to create a view like this one:
But that CREATE VIEW query fails. I believe this intermediate relation expects to be pointing a single file. Because there are multiple files broken out into multiple directories, I think the query should instead be something like:
Please let me know if my description is clear or if there's any additional info that would be helpful. Thanks!
The text was updated successfully, but these errors were encountered: