Boto3 does not work with openssl 2.x

[azachar]

Describe the bug

Hello,

The Databricks runtime 10.5 introduced an upgrade of openssl from 1.4.6 to 2.0.0, after that change, the latest boto3 library fails to install:

/databricks/python/lib/python3.8/site-packages/OpenSSL/crypto.py in <module>
   3222 
   3223 
-> 3224 utils.deprecated(
   3225     load_pkcs7_data,
   3226     __name__,

TypeError: deprecated() got an unexpected keyword argument 'name'

On databricks runtime 10.4 everything works just fine.

Could you please remove the deprecation issue?

Thank you!

Best regards,
Andrej

Expected Behavior

pip install boto3 pass

Current Behavior

pip install boto3 on databricks 10.5+ fails

Reproduction Steps

Open any databricks platform run a pip install boto3 on that platform.

Possible Solution

Upgrade openssl depedency

Additional Information/Context

No response

SDK version used

latest release

Environment details (OS name and version, etc.)

databricks 10.5

[tim-finnigan]

Hi @azachar thanks for reaching out. I think the error is due to using pyOpenSSL and boto3 in the same environment. There was a recent issue (boto/botocore#2744) that goes into more detail about the incompatibilities and deprecation.

[nettybun]

Ran into this and I'm not 100% sure how I fixed it but hoping this might help someone searching for keywords (as I was):

    import boto3
  File "/Users/hames/Library/Python/3.9/lib/python/site-packages/boto3/__init__.py", line 16, in <module>
    from boto3.session import Session
  File "/Users/hames/Library/Python/3.9/lib/python/site-packages/boto3/session.py", line 17, in <module>
    import botocore.session
  File "/Users/hames/Library/Python/3.9/lib/python/site-packages/botocore/session.py", line 30, in <module>
    import botocore.credentials
  File "/Users/hames/Library/Python/3.9/lib/python/site-packages/botocore/credentials.py", line 34, in <module>
    from botocore.config import Config
  File "/Users/hames/Library/Python/3.9/lib/python/site-packages/botocore/config.py", line 16, in <module>
    from botocore.endpoint import DEFAULT_TIMEOUT, MAX_POOL_CONNECTIONS
  File "/Users/hames/Library/Python/3.9/lib/python/site-packages/botocore/endpoint.py", line 22, in <module>
    from botocore.awsrequest import create_request_object
  File "/Users/hames/Library/Python/3.9/lib/python/site-packages/botocore/awsrequest.py", line 26, in <module>
    import botocore.utils
  File "/Users/hames/Library/Python/3.9/lib/python/site-packages/botocore/utils.py", line 33, in <module>
    import botocore.httpsession
  File "/Users/hames/Library/Python/3.9/lib/python/site-packages/botocore/httpsession.py", line 18, in <module>
    from urllib3.contrib.pyopenssl import orig_util_SSLContext as SSLContext
  File "/Users/hames/Library/Python/3.9/lib/python/site-packages/urllib3/contrib/pyopenssl.py", line 46, in <module>
    import OpenSSL.SSL
  File "/Library/Python/3.9/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import SSL, crypto
  File "/Library/Python/3.9/site-packages/OpenSSL/SSL.py", line 19, in <module>
    from OpenSSL.crypto import (
  File "/Library/Python/3.9/site-packages/OpenSSL/crypto.py", line 3224, in <module>
    utils.deprecated(
TypeError: deprecated() got an unexpected keyword argument 'name'

There's a mix of /Library/Python/3.9/site-packages/ and /Users/hames/Library/Python/3.9/lib/python/site-packages/ which is cause for concern. I read in boto/botocore#2744 (comment) that pyOpenSSL might cause problems. I had pyOpenSSL 22.1.0 in my pip list.

I removed everything in my system 😅 (be careful with this, but really, things should be venv'd or --user'd anyway)

pip freeze > requirements.txt
pip uninstall -r requirements.txt -y

After reinstalling my package (which uses boto3 as a dependency) I did pip list --user and there is no longer any pyOpenSSL ✨ and my package doesn't throw the error ✅.

TL;DR I think if you have pyOpenSSL that's bad and unnecessary (according to the linked GH thread).

[tim-finnigan]

I'm converting this issue to a Discussion rather than a bug because the problem involves pyOpenSSL and this is something that can't be controlled from the boto3 side. For more background please refer to the issue linked earlier. Removing pyOpenSSL as mentioned above should fix the issue.