Broken bundle install on AWS Ubuntu 16.04 via cloud-init

[MrLunar]

• I've gone though the User Guide and the API reference
• I've searched for previous similar issues and didn't find any solution

Describe the bug

Since 10th July (1.18.97 release), when launching a Ubuntu 16.04 instance on AWS EC2, attempting to install awscli within a User Data script will fail.

There appears to have been a recent change in the dependency constraint for rsa. The bundle now includes version 4.5 of this package. Unfortunately, this version appears to contain a regression from 4.4 within it's setup.py when no LANG environment variable is present to specify UTF-8 encoding.

During cloud-init scripts on Ubuntu 16.04s AMIs, there is no LANG environment variable. Thus setup.py attempts to decode a UTF-8 file as ASCII, which results in a fatal error.

This has resulted in instances now failing to launch.

A quick look at the rsa project appears to show this has been fixed in 4.6 with the re-addition of encoding='utf-8' to setup.py:
https://github.com/sybrenstuvel/python-rsa/compare/version-4.5...version-4.6

SDK version number
aws-cli/1.18.97 Python/3.5.2 Linux/4.4.0-1110-aws botocore/1.17.20

Platform/OS/Hardware/Device
Ubuntu 16.04 on AWS EC2 (ami-09623236df3ab2b4e)

To Reproduce (observed behavior)
Launch an AWS EC2 instance with a Ubuntu 16.04 AMI using the following User Data:

#!/bin/bash
apt-get update && apt-get install -y unzip python3-setuptools
curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
unzip awscli-bundle.zip
python3 ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws

(The same can also be simulated manually by prefixing the install command with LANG= )

Check for the error in /var/log/cloud-init-output.log (see output below).

Expected behavior
For the installation to complete successfully.

Logs/output

Running cmd: /usr/bin/python3 virtualenv.py --no-download --python /usr/bin/python3 /usr/local/aws
Running cmd: /usr/local/aws/bin/pip install --no-binary :all: --no-cache-dir --no-index --find-links file://. setuptools_scm-3.3.3.tar.gz
Running cmd: /usr/local/aws/bin/pip install --no-binary :all: --no-cache-dir --no-index --find-links file://. wheel-0.33.6.tar.gz
Running cmd: /usr/local/aws/bin/pip install --no-binary :all: --no-build-isolation --no-cache-dir --no-index  --find-links file:///awscli-bundle/packages awscli-1.18.97.tar.gz
Traceback (most recent call last):
  File "./awscli-bundle/install", line 218, in <module>
    main()
  File "./awscli-bundle/install", line 200, in main
    pip_install_packages(opts.install_dir)
  File "./awscli-bundle/install", line 138, in pip_install_packages
    pip_script, INSTALL_ARGS, PACKAGES_DIR, cli_tarball))
  File "./awscli-bundle/install", line 57, in run
    p.returncode, cmd, output))
__main__.BadRCError: Bad rc (1) for cmd '/usr/local/aws/bin/pip install --no-binary :all: --no-build-isolation --no-cache-dir --no-index  --find-links file:///awscli-bundle/packages awscli-1.18.97.tar.gz': Looking in links: file:///awscli-bundle/packages
Processing ./awscli-1.18.97.tar.gz
Processing ./botocore-1.17.20.tar.gz
Processing ./docutils-0.15.2.tar.gz
Processing ./s3transfer-0.3.3.tar.gz
Processing ./PyYAML-5.3.1.tar.gz
Processing ./colorama-0.4.3.tar.gz
Processing ./rsa-4.5.tar.gz
    ERROR: Command errored out with exit status 1:
     command: /usr/local/aws/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-h0gbqwtn/rsa/setup.py'"'"'; __file__='"'"'/tmp/pip-install-h0gbqwtn/rsa/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-h0gbqwtn/rsa/pip-egg-info
         cwd: /tmp/pip-install-h0gbqwtn/rsa/
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-h0gbqwtn/rsa/setup.py", line 20, in <module>
        long_description = f.read()
      File "/usr/local/aws/lib/python3.5/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 898: ordinal not in range(128)
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

[jekriske-lilly]

To add https://github.com/sybrenstuvel/python-rsa/blob/master/CHANGELOG.md

Regarding 4.6

No functional changes compared to version 4.2.

[kdaily]

Hi @MrLunar,

I gave this a try on the same AMI provided with the same User Data script. My installation was successful, here's the end of the cloud-init-output.log file:

Archive:  awscli-bundle.zip
  inflating: awscli-bundle/install
  inflating: awscli-bundle/packages/six-1.15.0.tar.gz
  inflating: awscli-bundle/packages/docutils-0.15.2.tar.gz
  inflating: awscli-bundle/packages/virtualenv-16.7.8.tar.gz
  inflating: awscli-bundle/packages/PyYAML-5.3.1.tar.gz
  inflating: awscli-bundle/packages/futures-3.3.0.tar.gz
  inflating: awscli-bundle/packages/pyasn1-0.4.8.tar.gz
  inflating: awscli-bundle/packages/rsa-3.4.2.tar.gz
  inflating: awscli-bundle/packages/awscli-1.18.99.tar.gz
  inflating: awscli-bundle/packages/urllib3-1.25.9.tar.gz
  inflating: awscli-bundle/packages/python-dateutil-2.8.0.tar.gz
  inflating: awscli-bundle/packages/jmespath-0.10.0.tar.gz
  inflating: awscli-bundle/packages/colorama-0.4.3.tar.gz
  inflating: awscli-bundle/packages/PyYAML-5.2.tar.gz
  inflating: awscli-bundle/packages/s3transfer-0.3.3.tar.gz
  inflating: awscli-bundle/packages/urllib3-1.25.7.tar.gz
  inflating: awscli-bundle/packages/setup/setuptools_scm-3.3.3.tar.gz
  inflating: awscli-bundle/packages/setup/wheel-0.33.6.tar.gz
Running cmd: /usr/bin/python3 virtualenv.py --no-download --python /usr/bin/python3 /usr/local/aws
Running cmd: /usr/local/aws/bin/pip install --no-binary :all: --no-cache-dir --no-index --find-links file://. setuptools_scm-3.3.3.tar.gz
Running cmd: /usr/local/aws/bin/pip install --no-binary :all: --no-cache-dir --no-index --find-links file://. wheel-0.33.6.tar.gz
Running cmd: /usr/local/aws/bin/pip install --no-binary :all: --no-build-isolation --no-cache-dir --no-index  --find-links file:///awscli-bundle/packages awscli-1.18.99.tar.gz
You can now run: /usr/local/bin/aws --version
Cloud-init v. 19.4-33-gbb4131a2-0ubuntu1~16.04.1 running 'modules:final' at Thu, 16 Jul 2020 19:37:46 +0000. Up 26.45 seconds.
Cloud-init v. 19.4-33-gbb4131a2-0ubuntu1~16.04.1 finished at Thu, 16 Jul 2020 19:39:18 +0000. Datasource DataSourceEc2Local.  Up 118.22 seconds

Of note, I see a different version of rsa installed. Are you installing other things that may have a different dependency?

[MrLunar]

Hi @kdaily

I'll have a closer look tomorrow, but I can see that there have been 2 awscli releases since this was raised (yours installed .99)

I can also see the following PR which has been merged since this was raised, so I wonder if this has (intentionally or not) fixed this issue? The PR pin version matches the one you installed.

#5379

Thanks.

[kdaily]

@MrLunar - looks like it, and thanks for seeing that! I meant to highlight that I tried the latest (1.18.99 just released). I'll try and verify what you saw on 1.18.97.

[MrLunar]

@kdaily Can confirm this is now working for me with 1.18.99 using the bundled rsa-3.4.2, same as your result 👍

Archive:  awscli-bundle.zip
  inflating: awscli-bundle/install
  inflating: awscli-bundle/packages/six-1.15.0.tar.gz
  inflating: awscli-bundle/packages/docutils-0.15.2.tar.gz
  inflating: awscli-bundle/packages/virtualenv-16.7.8.tar.gz
  inflating: awscli-bundle/packages/PyYAML-5.3.1.tar.gz
  inflating: awscli-bundle/packages/futures-3.3.0.tar.gz
  inflating: awscli-bundle/packages/pyasn1-0.4.8.tar.gz
  inflating: awscli-bundle/packages/rsa-3.4.2.tar.gz
  inflating: awscli-bundle/packages/awscli-1.18.99.tar.gz
  inflating: awscli-bundle/packages/urllib3-1.25.9.tar.gz
  inflating: awscli-bundle/packages/python-dateutil-2.8.0.tar.gz
  inflating: awscli-bundle/packages/jmespath-0.10.0.tar.gz
  inflating: awscli-bundle/packages/colorama-0.4.1.tar.gz
  inflating: awscli-bundle/packages/botocore-1.17.22.tar.gz
  inflating: awscli-bundle/packages/colorama-0.4.3.tar.gz
  inflating: awscli-bundle/packages/PyYAML-5.2.tar.gz
  inflating: awscli-bundle/packages/s3transfer-0.3.3.tar.gz
  inflating: awscli-bundle/packages/urllib3-1.25.7.tar.gz
  inflating: awscli-bundle/packages/setup/setuptools_scm-3.3.3.tar.gz
  inflating: awscli-bundle/packages/setup/wheel-0.33.6.tar.gz
+ python3 ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
Running cmd: /usr/bin/python3 virtualenv.py --no-download --python /usr/bin/python3 /usr/local/aws
Running cmd: /usr/local/aws/bin/pip install --no-binary :all: --no-cache-dir --no-index --find-links file://. setuptools_scm-3.3.3.tar.gz
Running cmd: /usr/local/aws/bin/pip install --no-binary :all: --no-cache-dir --no-index --find-links file://. wheel-0.33.6.tar.gz
Running cmd: /usr/local/aws/bin/pip install --no-binary :all: --no-build-isolation --no-cache-dir --no-index  --find-links file:///awscli-bundle/packages awscli-1.18.99.tar.gz
You can now run: /usr/local/bin/aws --version

[mahsoud]

Saw the same issue with awscli-1.18.99 and rsa-4.5
here is the log snippet:
https://gist.github.com/mahsoud/23324d89646c63c9ab3f3f8be98301f5

there was a similar issue: sybrenstuvel/python-rsa#129

[kdaily]

Hi @mahsoud, this should only affect the bundle. Are you still experiencing this?

[souschefistry]

@kdaily Can confirm this is now working for me with 1.18.99 using the bundled rsa-3.4.2, same as your result 👍

Archive:  awscli-bundle.zip
  inflating: awscli-bundle/install
  inflating: awscli-bundle/packages/six-1.15.0.tar.gz
  inflating: awscli-bundle/packages/docutils-0.15.2.tar.gz
  inflating: awscli-bundle/packages/virtualenv-16.7.8.tar.gz
  inflating: awscli-bundle/packages/PyYAML-5.3.1.tar.gz
  inflating: awscli-bundle/packages/futures-3.3.0.tar.gz
  inflating: awscli-bundle/packages/pyasn1-0.4.8.tar.gz
  inflating: awscli-bundle/packages/rsa-3.4.2.tar.gz
  inflating: awscli-bundle/packages/awscli-1.18.99.tar.gz
  inflating: awscli-bundle/packages/urllib3-1.25.9.tar.gz
  inflating: awscli-bundle/packages/python-dateutil-2.8.0.tar.gz
  inflating: awscli-bundle/packages/jmespath-0.10.0.tar.gz
  inflating: awscli-bundle/packages/colorama-0.4.1.tar.gz
  inflating: awscli-bundle/packages/botocore-1.17.22.tar.gz
  inflating: awscli-bundle/packages/colorama-0.4.3.tar.gz
  inflating: awscli-bundle/packages/PyYAML-5.2.tar.gz
  inflating: awscli-bundle/packages/s3transfer-0.3.3.tar.gz
  inflating: awscli-bundle/packages/urllib3-1.25.7.tar.gz
  inflating: awscli-bundle/packages/setup/setuptools_scm-3.3.3.tar.gz
  inflating: awscli-bundle/packages/setup/wheel-0.33.6.tar.gz
+ python3 ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
Running cmd: /usr/bin/python3 virtualenv.py --no-download --python /usr/bin/python3 /usr/local/aws
Running cmd: /usr/local/aws/bin/pip install --no-binary :all: --no-cache-dir --no-index --find-links file://. setuptools_scm-3.3.3.tar.gz
Running cmd: /usr/local/aws/bin/pip install --no-binary :all: --no-cache-dir --no-index --find-links file://. wheel-0.33.6.tar.gz
Running cmd: /usr/local/aws/bin/pip install --no-binary :all: --no-build-isolation --no-cache-dir --no-index  --find-links file:///awscli-bundle/packages awscli-1.18.99.tar.gz
You can now run: /usr/local/bin/aws --version

Downgrading to 1.18.99 using the bundled rsa-3.4.2 worked for me. Using Ubuntu 18.04 docker from Nvidia as base i.e. FROM nvcr.io/nvidia/cuda:11.2.1-devel-ubuntu18.04