Reduce Pydantic package size with no-binary flag when support is out

[zrothberg]

I was reading through the docs and saw that pydantic adds over 70 mbs. Poking around the pydantic repo there are a few threads about this subject. pydantic/pydantic#2276 pydantic/pydantic#2517. I was wondering if it was possible to use the --no-binary flag in your package level requirements.txt. I had seen this comment https://github.com/awslabs/aws-lambda-powertools-python/issues/481#issuecomment-863907997 and am unsure if the same issue that prevents the smaller cython binaries would prevent this from working. I also saw that poetry doesn't allow the --no-binary flag so wasn't sure if that could be a reason that preventing this from being recommended. Or if this works but wasn't mentioned in the docs due to speed concerns.

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.

[heitorlessa]

hey @zrothberg thanks for bringing this up! As you pointed out, there are two angles to this from what I could tell after some experimentation.

Publisher. Poetry didn't support the --no-binary flag (maybe it does now?). We'd have to experiment exporting extra dependencies as requirements from pyproject.toml, then install using --no-binary flag. I did find a workaround but found I needed more time to test it thoroughly since this project is used in production by hundreds of customers

Direct installs. SAM CLI doesn't support propagating pip flags, nor did propagate PIP flags via env var like PIP_NO_BINARY when using --use-container. I found that recommending customers to define those extra dependencies in requirements to be sub-optimal, as it can easily get out of date if we need to add more later.

Now that we have a reliable Layers infrastructure created by @am29d, maybe this could be worked around as a Lambda Layer since we have more control on how to package it (e.g. we could use a Docker Amazon Linux 2 + pip with no-binary flag).

For now, I'm actively following that Poetry issue as this provides a more reliable solution long-term. If you do find a better mechanism, by all means I'd love to get this improved indeed.

Thanks again!

[zrothberg]

@heitorlessa so I actually found a work around for this today. Appears someone on the workaround you commented on also found a similar thing. I was poking around the pip documentation and saw that it also allows you to specify the exact url to download a package from. If you use the py3-none-any.whl link from the pypi folder this will install without the binaries safely. I dropped 10 mbs off my zipped lambda and another 40mbs off my running size doing this. It's a reliable solution to the problem that doesn't cause weird issues. Only thing that may be annoying is dependabot won't be able to update dependencies automatically.

[heitorlessa]

cc @am29d ^^ o//. Though I think with the work of moving the Lambda Layer pipeline to AWS Code*, we can use pip flags to correctly address this without impacting dependabot upgrades.

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.

[heitorlessa]

assigning to @rubenfonseca as we've managed to make a HUGE improvement for V2, getting down to Pydantic fully optimized with Cython, yet ~2MB final Lambda Layer with ALL deps installed!

[rubenfonseca]

We are now close to release the v2 layer with all the optimizations mentioned on this issue, thank you so much for your great tips! You can check the current implementation on our CDK Layer construct: https://github.com/awslabs/cdk-aws-lambda-powertools-layer/blob/main/layer/Python/Dockerfile#L31-L47

[heitorlessa]

Closing as we're wrap to launch V2

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.