Add duckdb to packages

[cpcloud]

Description

Adds duckdb to the list of packages shipping with pyodide.

The included patches can be removed after the next duckdb release.

Checklists

• Add a CHANGELOG entry

[cpcloud]

It looks like the mypy build failed, not sure how to proceed.

[cpcloud]

cc @ryanking13 and @hoodmane, looks like y'all reviewed the previous attempt of duckdb.

[cpcloud]

Not sure how to get past the mypy build failure.

[hoodmane]

Well the mypy failure looks pretty unrelated to you. Try disabling it for now:

--- a/packages/mypy/meta.yaml
+++ b/packages/mypy/meta.yaml
@@ -1,6 +1,7 @@
 package:
   name: mypy
   version: 1.8.0
+  _disabled: true
   top-level:
     - mypyc
     - mypy

[cpcloud]

It looks like I might be hitting a timeout, the build just died after 1 hour.

[cpcloud]

Not entirely sure what the expectation is around build times here, but in the corresponding duckdb PR it takes 10-15 minutes to build one duckdb wasm wheel.

[cpcloud]

I'm happy to keep disabling packages until the duckdb package builds successfully, but that doesn't seem like a great solution.

[hoodmane]

Ah interesting. I guess we really need to figure out how to improve our package system at this point. I think @ryanking13 has thought about this more than I have. We could break the job up into two sets somehow...

[cpcloud]

Hopefully I'm not opening a can of worms here, but GitHub actions has a 6 hour timeout. Would moving these jobs to GHA be on the table?

[cpcloud]

It looks like there's a decent amount of circleci code to port in any case.

[hoodmane]

I think a lot of aspects of the CircleCI system is a lot better than github actions, including "rerun with ssh access," better handling of test results, the nice workflow chart showing the task DAG, etc. But a longer timeout would be useful. As I said, the CI and package build setup is something that @ryanking13 has been more directly working on of late so let's see what he has to say about it.

[ryanking13]

Hopefully I'm not opening a can of worms here, but GitHub actions has a 6 hour timeout. Would moving these jobs to GHA be on the table?

While this can be a temporary solution, we don't want to see CI running time continue to increase. A more fundamental solution is to move away from the way Pyodide currently builds all packages.

I think @ryanking13 has thought about this more than I have.

I'm working on separating the pyodide main module and package builds from this repository (https://github.com/ryanking13/pyodide-recipes-mirror). This requires some modifications to our build system (and distribution logic), so there's some work to be done.

While I'm working on that, as a temporary solution, we might consider pre-building some packages that take a very long time to build, but have no dependencies on other packages. mypy is one example (I think duckdb can be another candidate too), which takes 10+ minutes to build, but has no dependencies on other packages, making it easier to build and test separately.

[cpcloud]

Ah, ok. Great to hear. Is there anything I can do to help move this along?

[ryanking13]

Ah, ok. Great to hear. Is there anything I can do to help move this along?

It will probably take some time for me to work on our build system, so in the meantime I'd recommend building duckdb out-of-tree and see if it works well.

Here are some useful links you might find helpful

• Our documentation: https://pyodide.org/en/stable/development/building-and-testing-packages.html
• GHA workflow in Numpy: https://github.com/numpy/numpy/blob/main/.github/workflows/emscripten.yml

[cpcloud]

@ryanking13 Yep, last week a PR of mine was merged into DuckDB duckdb/duckdb#11531.

The wheels are building in CI there for both Python 3.10 and Python 3.11

I think it's really just about the CI issue here.