Quite long compilation time

[max-sixty]

Thanks for the excellent crate! We're using it in https://github.com/prql/prql for integration tests.

Something I wanted to highlight was the compilation time of the crate. At the moment, it's responsible for about 60% of our whole project's compilation time.

For a concrete case, check out the build timings here: https://github.com/prql/prql/suites/7516963360/artifacts/309340433 (just added in PRQL/prql#856 !).

Here's a screenshot:

Thanks!

[wangfenjin]

Thanks for the report. I did a quick check, one of the main reason is arrow crate takes too much time to build. It may not easy to optimize this, one option is switch to https://github.com/jorgecarleitao/arrow2 but I'm also not sure if that implementation is faster or not.

And I also noticed you use bundled feature of this crate, it will compile the c++ source code and also might takes too much time. I may suggest you to download the binary follow what I did in here https://github.com/wangfenjin/duckdb-rs/blob/main/.github/workflows/rust.yaml#L37-L51

[wangfenjin]

Asked in apache/arrow-rs#2170

[max-sixty]

Thanks a lot for looking into this @wangfenjin !

To update on this — I tried to understand why our cached build was taking so much time, given that duckdb-rs would have already been compiled. That's the most important metric for us, and that's the one that determines how long our CI takes.

I don't have a great answer, though this workflow shows that much of the time is coming from the prql-compiler integration test compilation, which is the one that uses duckdb-sys, and is otherwise very small. We would need to do deeper to confirm the cause (I'm not yet familiar in how to do that)

[digizeph]

With bundled flag, the duckdb 0.6.1 compile time is now over 4 minutes with my M1 Pro Macbook. I wonder if the recent upstream updates on the main branch has brought any significant improvements on the compile-time aspect?