Sharing Python code between catalog, ingestion_server and API

[obulat]

Problem

Some python code needs to be available in several parts of Openverse stack. Currently, we usually duplicate this kind of code. This sometimes causes it to by out of sync.

What is the best way of sharing this kind of code?
We use a separate package for Openverse linting configuration. Would we need to create a separate package for the common python code to be able to share it?

Additional information

Most of duplicated code is related to data cleaning:

• validating and fixing URLs
• validating license information
• validating media extensions
• tags cleanup
  Admittedly, some of these duplicates will be removed after the data normalization project is complete.

Existing issues

• Ingestion server does not check extracted extensions for validity #3827

[dhruvkb]

One subtle difference between the separate package for linting and this case is that the common code here is a regular runtime dependency and must also be included in our Docker images whereas the linting package is a dev-dependency that will not need to be included in the final Docker image.

[sarayourfriend]

PDM has monorepo functionality, which allows cross-package dependencies inside a monorepo. I've left a note about this on #286, but it's especially relevant for a question like this @dhruvkb. Probably also when thinking about how to bundle things into a docker image with only the runtime dependencies.

[dhruvkb]

I think we can close this as we have used PDM (in #4128 and #4107) to implement a mechanism for sharing code using libraries/sub-packages (not using monorepo) that can also optionally be distributed to PyPI if it is more generally useful. We also have #1009 to look into the separation of dependencies which would mean us only keeping runtime dependencies in the final image that will be pushed to GHCR.