How to determine required dependencies for new packages, and why and pyodide build needs metadata files for existing packages?

[bitsondatadev]

Hey everyone, I'm running a local Pyodide build on my Ubuntu box, Charizard, to create a new package for pyiceberg:

$ uname -a
Linux charizard 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

The local unchanged build w/Docker runs like a champ!

Followed these steps to build my meta.yaml file for PyIceberg:

1. Ran make from Docker deps container.

   brian@2395b805190f:/src$ pyodide skeleton pypi pyiceberg
   Creating meta.yaml package for pyiceberg
   packages/pyiceberg/meta.yaml 37ms
   Output written to /src/packages/pyiceberg/meta.yaml
   

2. Generated initial meta.yaml from PyPi and ran partial build with no deps.

   brian@2395b805190f:/src$ cat packages/pyiceberg/meta.yaml
   package:
     name: pyiceberg
     version: 0.6.0
     top-level:
       - hive_metastore
       - pyiceberg
       - fb303
   source:
     url: https://files.pythonhosted.org/packages/01/9e/417eb6ac44c9de920b5c360eb58075553ad6830a7425ea711b2742c3ba25/pyiceberg-0.6.0.tar.gz
     sha256: 0d965c202a9f49d0daa911c3fe061c55c90fa469bb382ce6440a7799ed35b003
   about:
     home: https://py.iceberg.apache.org/
     PyPI: https://pypi.org/project/pyiceberg
     summary: Apache Iceberg is an open table format for huge analytic datasets
     license: Apache-2.0
   extra:
     recipe-maintainers:
       - bitsondatadev
       
   brian@2395b805190f:/src$ PYODIDE_PACKAGES="pyiceberg" make
   
   ...
   
   [1/1] (thread 1) built pyiceberg in 10s
   
   Building packages... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/1 100% Time elapsed: 0:00:10
   Copying build logs to /src/packages/build-logs
   Copying built packages to /src/dist
   Writing pyodide-lock.json to /src/dist/pyodide-lock.json
   make[1]: Leaving directory '/src/packages'
   [2024-03-20 22:17:25] done building packages...
   

3. Based on the note in this section I looked up the meta.yaml in the pyiceberg conda-forge package and initially mapped all of those packages to my pyiceberg meta.yaml:

   $ cat packages/pyiceberg/meta.yaml
   package:
     name: pyiceberg
     version: 0.6.0
     top-level:
       - hive_metastore
       - pyiceberg
       - fb303
   #build:
     #script: python -m pip install . -vv --no-deps --no-build-isolation
   source:
     url: https://files.pythonhosted.org/packages/01/9e/417eb6ac44c9de920b5c360eb58075553ad6830a7425ea711b2742c3ba25/pyiceberg-0.6.0.tar.gz
     sha256: 0d965c202a9f49d0daa911c3fe061c55c90fa469bb382ce6440a7799ed35b003
   about:
     home: https://py.iceberg.apache.org/
     PyPI: https://pypi.org/project/pyiceberg
     summary: Apache Iceberg is an open table format for huge analytic datasets
     license: Apache-2.0
   requirements:
     host:
     # Dependencies that are needed to build the package
       #- poetry-core >=1.0.0
       - wheel
       #- cython >=3.0.0
       #- setuptools
       #- pip
     #run:
     # Dependencies that are needed to run the package
       #- mmhash3 >=3.0.0,<4.0.0
       #- mmh3 >=3.0.0,<4.0.0     # pip uses mmhash3
       #- requests >=2.20.0,<3.0.0
       #- click >=7.1.1,<9.0.0
       #- rich >=10.11.0,<14.0.0
       #- strictyaml >=1.7.0,<2.0.0
       #- pydantic >=2.0,<3.0,!=2.4.0,!=2.4.1
       #- sortedcontainers ==2.4.0
       #- fsspec >=2023.1.0,<2024.1.0
       #- pyparsing >=3.1.0,<4.0.0
       #- zstandard >=0.13.0,<1.0.0
       #- pyarrow >=9.0.0,<16.0.0
       #- pandas >=1.0.0,<3.0.0
       #- duckdb >=0.5.0,<1.0.0
       #- ray >=2.0.0,<2.10.0
       #- python-snappy >=0.6.0,<1.0.0
       #- thrift >=0.13.0,<1.0.0
       #- mypy-boto3-glue >=1.28.18
       #- boto3 >=1.24.59
       #- s3fs >=2023.1.0,<2024.1.0
       #- adlfs >=2023.1.0,<2024.3.0
       #- gcsfs >=2023.1.0,<2024.1.0
       #- psycopg2-binary >=2.9.6
       #- sqlalchemy >=1.4.0,<3.0.0
   

   But unfortunately each of these libraries (in this example I have wheel uncommented), I've tried a bunch of these in isolation and I keep getting an error message for all of these libraries:

   ValueError: No metadata file found for the following package: wheel
   make[1]: *** [Makefile:8: all] Error 1
   make[1]: Leaving directory '/src/packages'
   make: *** [Makefile:278: dist/pyodide-lock.json] Error 2
   

   What is strange for me is that I get these errors for like pydantic, sqlalchemy, pandas, requests, and many others that already exist in the official Pyodide packages directory.

With that context, I imagine I'm missing something regarding how to connect dependencies for the ones that should already be included, but I would have thought those would be created and linked in some way during the main build.

1. I have the following questions around building packages and their dependencies.

   1. ✅ ${\color{lightgreen}Answered}$ What is causing the issue with any and all of these dependencies that already have a Pyodide package throwing this no metadata file error?
      Answer:

      • https://github.com/pyodide/pyodide/discussions/4632#discussioncomment-8875388
      • https://github.com/pyodide/pyodide/discussions/4632#discussioncomment-8869904
      • https://github.com/pyodide/pyodide/discussions/4632#discussioncomment-8870040

   2. ✅ ${\color{lightgreen}Answered}$ For the packages that don't exist, do I need to create my own project for each of those? For example mmh3/mmhash3 or s3fs.
      Answer:

      • https://github.com/pyodide/pyodide/discussions/4632#discussioncomment-8875516
      • The answer to this is yes, I would need to create this project only for the projects that aren't pure python (mmh3 in my example).

   3. ✅ ${\color{lightgreen}Answered}$ How do I know if something is pure python or not? I get it shouldn't have C stuff if it's pure but how do you quickly and accurately determine that (taking s3fs for example)?
      Answer:

      • https://github.com/pyodide/pyodide/discussions/4632#discussioncomment-8875423
      • https://github.com/pyodide/pyodide/discussions/4632#discussioncomment-8870681

2. ✅ ${\color{lightgreen}Answered}$ Something else I've noticed, I don't see many other packages installing this many libraries so I assume I'm inherently not understanding which dependencies need to be listed here, and which don't. Could somebody guide me on how to know which dependencies are required and which ones aren't? For example it's strange that I have to list python, pip, wheel, etc...
   Answer:

   • https://github.com/pyodide/pyodide/discussions/4632#discussioncomment-8875516

3. ✅ ${\color{lightgreen}Answered}$ If these bigger packages that are being worked on like pyarrow and duckdb are run dependencies (not the build one) then we can still install and package the project, just need to avoid installing those sub packages?
   Answer:

   • https://github.com/pyodide/pyodide/discussions/4632#discussioncomment-8875516

I promise I will submit some docs on this for the next poor uneducated fool like me if you help me. I was very careful to craft this question as I will use this to kind of guide the docs I will add.

Thanks for listening to my TED squalk!

[bitsondatadev]

Okay, already answered question 1.i:

If PYODIDE_PACKAGES is not set, a minimal set of packages necessary to run the core test suite is installed, including “micropip”, “pyparsing”, “pytz”, “packaging”, “Jinja2”, “regex”. This is equivalent to setting PYODIDE_PACKAGES='tag:core' meta-package. Other supported meta-packages...
“*” builds all packages, prefixing a package with "!" excludes it and micropip and distutils are always automatically included.

So the just pull everything answer would be to run the following but it's currently 230 packages which would take some time to build:

PYODIDE_PACKAGES="*" make

Still not sure how I determine if something is pure python once I run this to determine the rest of the projects I would actually need to include.

Note: add a bold header over this section of the docs "Building existing Pyodide Packages". The current docs for this makes it look a bit optional and I grazed right over that.

[ryanking13]

What is causing the issue with any and all of these dependencies that already have a Pyodide package throwing this no metadata file error?

You don't need to build all packages for that. Pyodide will use the requirements section in the meta.yaml file to find which packages it needs to build.

ValueError: No metadata file found for the following package: wheel

This error message happens because you listed wheel in requirements/host, but there is not meta.yaml file for wheel. But actually you don't need to add wheel to requirements/host as it is already added in the pyproject.toml of pyiceberg

[bitsondatadev]

Just figured out that you shouldn't put the version ranges in meta.yaml of Pyodide. As mentioned in the docs conda-forge meta.yaml has more syntax and Pyodide currently only requires the package name/tag.

So - pydantic >=2.0,<3.0,!=2.4.0,!=2.4.1 should just be - pydantic.

This was specifically what was causing the error with referencing package name that didn't exist.

[bitsondatadev]

After removing the version ranges and uncommenting the package dependencies that do exist in Pyodide already, I have a successful build. Here's what my meta.yaml looks like now:

cat packages/pyiceberg/meta.yaml
package:
  name: pyiceberg
  version: 0.6.0
  top-level:
    - hive_metastore
    - pyiceberg
    - fb303
#build:
  #script: python -m pip install . -vv --no-deps --no-build-isolation
source:
  url: https://files.pythonhosted.org/packages/01/9e/417eb6ac44c9de920b5c360eb58075553ad6830a7425ea711b2742c3ba25/pyiceberg-0.6.0.tar.gz
  sha256: 0d965c202a9f49d0daa911c3fe061c55c90fa469bb382ce6440a7799ed35b003
about:
  home: https://py.iceberg.apache.org/
  PyPI: https://pypi.org/project/pyiceberg
  summary: Apache Iceberg is an open table format for huge analytic datasets
  license: Apache-2.0
requirements:
  host:
  # Dependencies that are needed to build the package
    - click
    - fsspec
    #- mmh3 # pip uses mmhash3
    #- mmhash3
    - pydantic
    - pyparsing
    - requests
    #- rich
    #- sortedcontainers
    #- strictyaml
    #- zstandard
  run:
  # Dependencies that are needed to run the package
    #- pyarrow
    #- thrift
    #- mypy-boto3-glue
    #- python-snappy
    #- ray
    #- adlfs
    #- gcsfs
    #- psycopg2-binary
    #- s3fs
    - pandas
    #- boto3
    - sqlalchemy
    #- duckdb

I have also split the packages into host vs run requirements based on the Dependency vs Constraint split I see on the pyiceberg conda-forge package.

The files that are commented out are the files that pyodide doesn't currently support. Once we answer question 1.ii and 1.iii, then I can figure out which ones I likely need to also add support for to at least get the build working. Which if all my assumtions are correct about the build, would be, mmh3, mmhash3, rich, sortedcontainers, strictyaml, and zstandard assuming none of those are pure python.

[bitsondatadev]

So regarding to tell if something is a pure python function, I found this community SO answer that seems relatively smart and over my head so I'm just trusting it:

isitc.py

import inspect, os
from importlib.machinery import ExtensionFileLoader, EXTENSION_SUFFIXES
from types import ModuleType

def is_c_extension(module: ModuleType) -> bool:
    '''
    `True` only if the passed module is a C extension implemented as a
    dynamically linked shared library specific to the current platform.

    Parameters
    ----------
    module : ModuleType
        Previously imported module object to be tested.

    Returns
    ----------
    bool
        `True` only if this module is a C extension.
    '''
    assert isinstance(module, ModuleType), '"{}" not a module.'.format(module)

    # If this module was loaded by a PEP 302-compliant CPython-specific loader
    # loading only C extensions, this module is a C extension.
    if isinstance(getattr(module, '__loader__', None), ExtensionFileLoader):
        return True

    # Else, fallback to filetype matching heuristics.
    #
    # Absolute path of the file defining this module.
    module_filename = inspect.getfile(module)

    # "."-prefixed filetype of this path if any or the empty string otherwise.
    module_filetype = os.path.splitext(module_filename)[1]

    # This module is only a C extension if this path's filetype is that of a
    # C extension specific to the current platform.
    return module_filetype in EXTENSION_SUFFIXES

Then run this against the libraries in question:

import mmh3, rich, sortedcontainers, strictyaml, zstandard
from isitc import is_c_extension
for module in (mmh3, rich, sortedcontainers, strictyaml, zstandard):
     print('Is "{}" a C extension? {}'.format(
         module.__name__, is_c_extension(module)))
         

Is "mmh3" a C extension? True
Is "rich" a C extension? False
Is "sortedcontainers" a C extension? False
Is "strictyaml" a C extension? False
Is "zstandard" a C extension? False

So in theory, I should only need to support the mmh3 package along with pyiceberg to get the build going. Luckily that package already has wheel distributions on pypi.

So I'm still not sure about the answer to 1.ii, but this seems to be a decent way to determine if it's a pure python package. I'm sure there's an easier way, but this comes from someone who knows little about python packaging stuff.

[ryanking13]

If the wheel filename ends with py3-none.any.whl, than it is a pure python wheel (which means that it does not contain platform specific binaries and it only contains Python files in it). Non-pure-python wheels have names such as manylinux_2_17_aarch64.manylinux2014_aarch64.whl. You can check the download section for each package in PyPI.

[bitsondatadev]

Wonderful :) I thought that would be too simple and there must be more nuance to it. Thanks @ryanking13!

[ryanking13]

Something else I've noticed, I don't see many other packages installing this many libraries so I assume I'm inherently not understanding which dependencies need to be listed here, and which don't. Could somebody guide me on how to know which dependencies are required and which ones aren't?

There are multiple ways to check the dependencies for each package.

• pip download <package>: this command will download all dependencies of the package recursively so you can check which packages are dependencies of the package.
• check the METADATA file inside the wheel file of the package. Each wheel file has a file named METADATA in it, and it lists the dependencies of the package. For instance, pyiceberg-0.6.0 has following dependencies listed in the wheel file.

Requires-Dist: adlfs (>=2023.1.0,<2024.3.0) ; extra == "adlfs"
Requires-Dist: boto3 (>=1.24.59) ; extra == "glue" or extra == "dynamodb"
Requires-Dist: click (>=7.1.1,<9.0.0)
Requires-Dist: duckdb (>=0.5.0,<1.0.0) ; extra == "duckdb"
Requires-Dist: fsspec (>=2023.1.0,<2024.1.0)
Requires-Dist: gcsfs (>=2023.1.0,<2024.1.0) ; extra == "gcsfs"
Requires-Dist: getdaft (>=0.2.12) ; extra == "daft"
Requires-Dist: mmhash3 (>=3.0.0,<4.0.0)
Requires-Dist: mypy-boto3-glue (>=1.28.18) ; extra == "glue"
Requires-Dist: pandas (>=1.0.0,<3.0.0) ; extra == "pandas" or extra == "ray"
Requires-Dist: psycopg2-binary (>=2.9.6) ; extra == "sql-postgres"
Requires-Dist: pyarrow (>=9.0.0,<16.0.0) ; extra == "pyarrow" or extra == "pandas" or extra == "duckdb" or extra == "ray"
Requires-Dist: pydantic (>=2.0,<3.0,!=2.4.0,!=2.4.1)
Requires-Dist: pyparsing (>=3.1.0,<4.0.0)
Requires-Dist: python-snappy (>=0.6.0,<1.0.0) ; extra == "snappy"
Requires-Dist: ray (>=2.0.0,<2.10.0) ; extra == "ray"
Requires-Dist: requests (>=2.20.0,<3.0.0)
Requires-Dist: rich (>=10.11.0,<14.0.0)
Requires-Dist: s3fs (>=2023.1.0,<2024.1.0) ; extra == "s3fs"
Requires-Dist: sortedcontainers (==2.4.0)
Requires-Dist: sqlalchemy (>=2.0.18,<3.0.0) ; extra == "sql-postgres" or extra == "sql-sqlite"
Requires-Dist: strictyaml (>=1.7.0,<2.0.0)
Requires-Dist: thrift (>=0.13.0,<1.0.0) ; extra == "hive"
Requires-Dist: zstandard (>=0.13.0,<1.0.0) ; extra == "zstandard"

Looking at the METADATA file is useful as you can check which packages are mandatory and which ones are optional. For instance, the package ray is optional for pyiceberg. So you don't need it unless you really need to use it (same for duckdb, pyarrow, daft, etc).

For example it's strange that I have to list python, pip, wheel, and

You don't have to list python, pip, etc. Though Pyodide's meta.yaml file is inspired by conda's meta.yaml, but it's not identical. You only need to list Python package dependencies (+ some libraries if the package requires it).

[ryanking13]

If these bigger packages that are being worked on like pyarrow and duckdb are run dependencies (not the build one) then we can still install and package the project, just need to avoid installing those sub packages?

This question is also answered in this comment. You don't need to build extra dependencies unless you use it.