charm-python-packages does not guarantee these packages are installed before packages in requirements.txt

[DnPlas]

Bug Description

I have noticed that the list of packages in charm-python-packages may not guarantee they will be installed before charm requirements, as the docs suggest.

The behaviour I am getting is that the packages listed in charm-python-packages get collected and it looks like they are being installed as the very first thing, for example:

:: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=anyio,attrs,certifi,charmed-kubeflow-chisme,charset-normalizer,deepdiff,exceptiongroup,h11,httpcore,httpx,idna,importlib-resources,jinja2,jsonschema,lightkube,lightkube-models,markupsafe,oci-image,ops,ordered-set,pip,pkgutil-resolve-name,pyrsistent,pyyaml,requests,ruamel-yaml,ruamel-yaml-clib,serialized-data-interface,setuptools,sniffio,tenacity,urllib3,websocket-client,wheel,zipp', '--requirement=requirements.txt', 'pip>23.0', 'setuptools>69.0', 'wheel']    
::    :: Collecting pip>23.0                                                                                                                                                                 
::    ::   Using cached pip-24.0.tar.gz (2.1 MB)                                                                                                                                             
::    ::   Installing build dependencies: started                                                                                                                                            
::    ::   Installing build dependencies: finished with status 'done'                                                                                                                        
::    ::   Getting requirements to build wheel: started                                                                                                                                      
::    ::   Getting requirements to build wheel: finished with status 'done'                                                                                                                  
::    ::     Preparing wheel metadata: started                                                                                                                                               
::    ::     Preparing wheel metadata: finished with status 'done'                                                                                                                           
::    :: Collecting setuptools>69.0                                                                                                                                                          
::    ::   Using cached setuptools-69.5.1.tar.gz (2.3 MB)                                                                                                                                    
::    ::   Getting requirements to build wheel: started                                                                                                                                      
::    ::   Getting requirements to build wheel: finished with status 'done'                                                                                                                  
::    ::   Installing backend dependencies: started                                                                                                                                          
::    ::   Installing backend dependencies: finished with status 'done'                                                                                                                      
::    ::     Preparing wheel metadata: started                                                                                                                                               
::    ::     Preparing wheel metadata: finished with status 'done' 
::    :: Collecting jinja2==3.1.2                                                                                                                                                            
::    ::   Using cached Jinja2-3.1.2.tar.gz (268 kB)                                                                                                                                         

but in reality they get installed after collecting all packages, including the ones listed in requirements.txt.

# There is a long list of `::    :: Collecting <package-name>...` before the following lines
::    :: Successfully built wheel setuptools pip                                                                                                                                             
::    :: Installing collected packages: wheel, setuptools, pip                                                                                                                               
::    ::   Attempting uninstall: setuptools                                                                                                                                                  
::    ::     Found existing installation: setuptools 44.0.0                                                                                                                                  
::    ::     Uninstalling setuptools-44.0.0:                                                                                                                                                 
::    ::       Successfully uninstalled setuptools-44.0.0                                                                                                                                    
::    ::   Attempting uninstall: pip                                                                                                                                                         
::    ::     Found existing installation: pip 20.0.2                                                                                                                                         
::    ::     Uninstalling pip-20.0.2:                                                                                                                                                        
::    ::       Successfully uninstalled pip-20.0.2     

This could result in errors if any of the charm requirements depends on a charm-python-package to be installed (or upgraded) a priori.

Please NOTE I concluded this was an issue after trying to solve this other issue with the suggestions from this post. Upgrading setuptools fixes the issue with markupsafe and jinja2, I tested it in a virtual env, but it was not possible in the charm.

To Reproduce

1. Take this charm.
2. charmcraft pack -v
3. Check the build logs and observe when the setuptools and pip packages get actually installed and upgraded.

Environment

• Ubuntu 20.04
• charmcraft latest/candidate

charmcraft.yaml

type: "charm"
bases:
  - build-on:
    - name: "ubuntu"
      channel: "20.04"
    run-on:
    - name: "ubuntu"
      channel: "20.04"
parts:
  charm:
    charm-python-packages: [setuptools>69.0, pip>23.0]  # Fixes install of some packages

Relevant log output

Logs: https://pastebin.canonical.com/p/DhpBRNn7pz/

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CRAFT-2848.

This message was autogenerated

[DnPlas]

Another way in which this issue could be reproduced is by running pip install commands in a virtual environment directly and observing what happens first. I have this:

ubuntu@charm-dev-focal:~$ source jinja2-test/bin/activate
(jinja2-test) ubuntu@charm-dev-focal:~$ pip list | grep setuptools
setuptools    44.0.0 
(jinja2-test) ubuntu@charm-dev-focal:~$ pip install --no-binary=jinja2,pip,setuptools --requirement=requirements.txt pip>23.0 setuptools>69.0 wheel # <--- this is the command that gets run during a build
    ERROR: Command errored out with exit status 1:
     command: /home/ubuntu/jinja2-test/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-pa0uqgo3/jinja2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-pa0uqgo3/jinja2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-pa0uqgo3/jinja2/pip-egg-info
         cwd: /tmp/pip-install-pa0uqgo3/jinja2/
    Complete output (35 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-pa0uqgo3/jinja2/setup.py", line 4, in <module>
        setup(
      File "/home/ubuntu/jinja2-test/lib/python3.8/site-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib/python3.8/distutils/core.py", line 121, in setup
        dist.parse_config_files()
      File "/home/ubuntu/jinja2-test/lib/python3.8/site-packages/setuptools/dist.py", line 701, in parse_config_files
        parse_configuration(self, self.command_options,
      File "/home/ubuntu/jinja2-test/lib/python3.8/site-packages/setuptools/config.py", line 121, in parse_configuration
        meta.parse()
      File "/home/ubuntu/jinja2-test/lib/python3.8/site-packages/setuptools/config.py", line 426, in parse
        section_parser_method(section_options)
      File "/home/ubuntu/jinja2-test/lib/python3.8/site-packages/setuptools/config.py", line 399, in parse_section
        self[name] = value
      File "/home/ubuntu/jinja2-test/lib/python3.8/site-packages/setuptools/config.py", line 184, in __setitem__
        value = parser(value)
      File "/home/ubuntu/jinja2-test/lib/python3.8/site-packages/setuptools/config.py", line 515, in _parse_version
        version = self._parse_attr(value, self.package_dir)
      File "/home/ubuntu/jinja2-test/lib/python3.8/site-packages/setuptools/config.py", line 349, in _parse_attr
        module = import_module(module_name)
      File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
      File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
      File "<frozen importlib._bootstrap>", line 991, in _find_and_load
      File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 848, in exec_module
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "/tmp/pip-install-pa0uqgo3/jinja2/src/jinja2/__init__.py", line 8, in <module>
        from .environment import Environment as Environment
      File "/tmp/pip-install-pa0uqgo3/jinja2/src/jinja2/environment.py", line 14, in <module>
        from markupsafe import Markup
    ModuleNotFoundError: No module named 'markupsafe'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.