installation in development conda env fails #893

lee1043 · 2023-01-31T22:29:37Z

What happened:

Installation in development conda env fails with latest setuptools.

To reproduce error:

1. Generate development conda env following this

conda env create -f conda-env/dev.yml -n my_pmp_conda_env
conda activate my_pmp_conda_env

2. From PMP local directory cloned from github

python setup.py install

Error:

File existing, purging: DefArgsCIA.json
/home/lee1043/.conda/envs/pcmdi_metrics_dev_20230131/lib/python3.9/site-packages/setuptools/dist.py:548: UserWarning: The version specified ('v2.5.1-18-gd0775e2') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.
  warnings.warn(
running install
/home/lee1043/.conda/envs/pcmdi_metrics_dev_20230131/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/home/lee1043/.conda/envs/pcmdi_metrics_dev_20230131/lib/python3.9/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
Traceback (most recent call last):
  File "/home/lee1043/git/pcmdi_metrics_20221013/pcmdi_metrics/setup.py", line 144, in <module>
    setup(
  File "/home/lee1043/.conda/envs/pcmdi_metrics_dev_20230131/lib/python3.9/site-packages/setuptools/__init__.py", line 108, in setup
    return distutils.core.setup(**attrs)
  File "/home/lee1043/.conda/envs/pcmdi_metrics_dev_20230131/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
  File "/home/lee1043/.conda/envs/pcmdi_metrics_dev_20230131/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
  File "/home/lee1043/.conda/envs/pcmdi_metrics_dev_20230131/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
    self.run_command(cmd)
  File "/home/lee1043/.conda/envs/pcmdi_metrics_dev_20230131/lib/python3.9/site-packages/setuptools/dist.py", line 1213, in run_command
    super().run_command(command)
  File "/home/lee1043/.conda/envs/pcmdi_metrics_dev_20230131/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/home/lee1043/.conda/envs/pcmdi_metrics_dev_20230131/lib/python3.9/site-packages/setuptools/command/install.py", line 74, in run
    self.do_egg_install()
  File "/home/lee1043/.conda/envs/pcmdi_metrics_dev_20230131/lib/python3.9/site-packages/setuptools/command/install.py", line 117, in do_egg_install
    cmd.ensure_finalized()  # finalize before bdist_egg munges install cmd
  File "/home/lee1043/.conda/envs/pcmdi_metrics_dev_20230131/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 111, in ensure_finalized
    self.finalize_options()
  File "/home/lee1043/.conda/envs/pcmdi_metrics_dev_20230131/lib/python3.9/site-packages/setuptools/command/easy_install.py", line 311, in finalize_options
    self.local_index = Environment(self.shadow_path + sys.path)
  File "/home/lee1043/.conda/envs/pcmdi_metrics_dev_20230131/lib/python3.9/site-packages/pkg_resources/__init__.py", line 1044, in __init__
    self.scan(search_path)
  File "/home/lee1043/.conda/envs/pcmdi_metrics_dev_20230131/lib/python3.9/site-packages/pkg_resources/__init__.py", line 1077, in scan
    self.add(dist)
  File "/home/lee1043/.conda/envs/pcmdi_metrics_dev_20230131/lib/python3.9/site-packages/pkg_resources/__init__.py", line 1096, in add
    dists.sort(key=operator.attrgetter('hashcmp'), reverse=True)
  File "/home/lee1043/.conda/envs/pcmdi_metrics_dev_20230131/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2631, in hashcmp
    self.parsed_version,
  File "/home/lee1043/.conda/envs/pcmdi_metrics_dev_20230131/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2685, in parsed_version
    raise packaging.version.InvalidVersion(f"{str(ex)} {info}") from None
pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: 'v3.0-46-ge6f562d' (package: cdutil)

Temporary solution

1. Check `setuptools` version and downgrade setuptools

>>> conda list setuptools
setuptools                66.1.1             pyhd8ed1ab_0    conda-forge

>>> pip install setuptools==65.5.0

2. Check if working: Repeat `python setup.py install` that seems work for main branch

2-1. main branch

mean_climate_driver.py --help works.

2-2. development branch

git checkout 859_pjg_xcdatclims_ljw_replace
python setup.py install

mean_climate_driver.py --help works.

Options for resolving

[1] Need to resolve setuptools issue
[2] fix setuptools version from dev.yml during the mean time
[3] Or change setuptools to pip

The text was updated successfully, but these errors were encountered:

lee1043 · 2023-02-01T19:12:21Z

@tomvothecoder This issue is the setuptools issue in PMP that I mentioned during the meeting this morning. Would you be able to share your knowledge on this? In particular, I wonder if I can get your help on establishing/enabling pip installation process for developmental conda env that was generated by using dev.yml (e.g., conda env create -f conda-env/dev.yml -n my_pmp_conda_env). Thank you for your attention and consideration.

tomvothecoder · 2023-02-02T21:53:08Z

Hi @lee1043, I found a few related issues and comments:

InvalidVersion exception when invalid version used on Setuptools 66 pypa/setuptools#3772
InvalidVersion exception when invalid version used on Setuptools 66 pypa/setuptools#3772 (comment)
- Mentions: Remove support for LegacyVersion pypa/setuptools#2497

setuptools>=66 now enforces PEP440 which breaks a ton of packages (such as cdutil) that do not conform to this versioning standard.

I didn't find an obvious solution to this issue besides packages needing to conform to this standard.
It looks like this affects python setup.py install and pip install ..

(pmp_dev) bash-4.2$ python -m pip install .
Processing /home/vo13/PCMDI/pcmdi_metrics
  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /home/vo13/miniconda3/envs/pmp_dev/bin/python /home/vo13/miniconda3/envs/pmp_dev/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmp8y38ymq6
       cwd: /tmp/pip-req-build-fcz5fyl5
  Complete output (39 lines):
  File existing, purging: DefArgsCIA.json
  /tmp/pip-build-env-xmzgfa23/overlay/lib/python3.9/site-packages/setuptools/dist.py:548: UserWarning: The version specified ('v2.5.1-18-gd0775e26') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.
    warnings.warn(
  running egg_info
  Traceback (most recent call last):
    File "/home/vo13/miniconda3/envs/pmp_dev/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 349, in <module>
      main()
    File "/home/vo13/miniconda3/envs/pmp_dev/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 331, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/vo13/miniconda3/envs/pmp_dev/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 117, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-xmzgfa23/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
    File "/tmp/pip-build-env-xmzgfa23/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-xmzgfa23/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 484, in run_setup
      super(_BuildMetaLegacyBackend,
    File "/tmp/pip-build-env-xmzgfa23/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 335, in run_setup
      exec(code, locals())
    File "<string>", line 144, in <module>
    File "/tmp/pip-build-env-xmzgfa23/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 108, in setup
      return distutils.core.setup(**attrs)
    File "/tmp/pip-build-env-xmzgfa23/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
    File "/tmp/pip-build-env-xmzgfa23/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/tmp/pip-build-env-xmzgfa23/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/tmp/pip-build-env-xmzgfa23/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 1213, in run_command
      super().run_command(command)
    File "/tmp/pip-build-env-xmzgfa23/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
      cmd_obj.ensure_finalized()
    File "/tmp/pip-build-env-xmzgfa23/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 111, in ensure_finalized
      self.finalize_options()
    File "/tmp/pip-build-env-xmzgfa23/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 219, in finalize_options
      parsed_version = parse_version(self.egg_version)
    File "/tmp/pip-build-env-xmzgfa23/overlay/lib/python3.9/site-packages/pkg_resources/_vendor/packaging/version.py", line 197, in __init__
      raise InvalidVersion(f"Invalid version: '{version}'")
  pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: 'v2.5.1-18-gd0775e26'
  ----------------------------------------
WARNING: Discarding file:///home/vo13/PCMDI/pcmdi_metrics. Command errored out with exit status 1: /home/vo13/miniconda3/envs/pmp_dev/bin/python /home/vo13/miniconda3/envs/pmp_dev/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmp8y38ymq6 Check the logs for full command output.
ERROR: Command errored out with exit status 1: /home/vo13/miniconda3/envs/pmp_dev/bin/python /home/vo13/miniconda3/envs/pmp_dev/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmp8y38ymq6 Check the logs for full command output.
(pmp_dev) bash-4.2$   pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: 'v2.5.1-18-gd0775e26'
bash: pkg_resources.extern.packaging.version.InvalidVersion:: command not found
(pmp_dev) bash-4.2$

The temporary workaround is pin setuptools<66.

I am looking into replacing local build installation processes with conda build for xCDAT and E3SM tools. One this is implemented, you can try copying the same process for pcmdi_metrics.

tomvothecoder · 2023-02-02T21:54:17Z

Hey, @jasonb5 and @xylar do either of you have any thoughts here?

xylar · 2023-02-03T03:34:40Z

@lee1043 and @tomvothecoder, does it help if you do this

python -m pip install .

instead of this?

python setup.py install

The latter is not really considered a good practice these days, I think. If dependencies are missing that are listed in setup.py when you build the development conda environment, that might also explain the trouble. These are just guesses. I can try this out myself if my suggestion above doesn't help.

xylar · 2023-02-03T03:40:17Z

Sorry, I take that back. If you download the cdutil conda package, you can open it up and see that invalid version string: v3.0-46-ge6f562d, which is nothing to do with the actual version (8.2.1). I think this is a bug in cdutil itself.

xylar · 2023-02-03T04:10:05Z

I think rebuilding the cdutil package will fix the issue. It's at least worth a try. A local build seemed to have a valid version number. However, testing failed because of an unrelated numpy.float issue in cdms2 that I'm attempting to fix in:
conda-forge/cdms2-feedstock#81

The version number stored internally in the conda package came out as: `v3.0-46-ge6f562d` This caused issues in setting up downstream packages: PCMDI/pcmdi_metrics#893 Hopefully, a rebuild helps.

xylar · 2023-02-03T04:23:08Z

This problem is really cascading. Have to fix the same numpy problem in `distarray:
conda-forge/distarray-feedstock#5

xylar · 2023-02-03T06:44:25Z

I'm making progress on this but haven't finished and will continue tomorrow.

lee1043 · 2023-02-03T06:49:21Z

@tomvothecoder @xylar thank you for stepping in and sharing your knowledge and experience!

The temporary workaround is pin setuptools<66.

@tomvothecoder I agree this would be the workaround for now but would be interested to learn more about more efficient way!

@xylar I appreciate your work for updating the cdutil version. I wonder if that going to be only cdutil or also other cdat packages such as genutil, cdms, etc.

xylar · 2023-02-03T06:50:30Z

I have to rebuild cdms2 and distarray anyways. If other packages cause trouble, we'll just rebuild them, too.

xylar · 2023-02-03T07:22:45Z

The weird version number must come in through this:
https://github.com/CDAT/cdutil/blob/v8.2.1/setup.py#L13-L17
The thing I don't understand is why the command wouldn't fail, since the source isn't in a git repo.

xylar · 2023-02-03T07:41:13Z

@lee1043 and @tomvothecoder, I think you could make headway by removing the version weirdness here.
Change:
https://github.com/PCMDI/pcmdi_metrics/blob/main/setup.py#L16
to

release_version = '2.5.1'

change:
https://github.com/PCMDI/pcmdi_metrics/blob/main/setup.py#L29
to

    descr = release_version

and finally change:
https://github.com/PCMDI/pcmdi_metrics/blob/main/setup.py#L146
to

    version=release_version,

This way, you can still save the git provenance in pcmdi_metrics/version.py in case that's useful but you don't try to use the git hash as a version number for the package itself.

xylar · 2023-02-03T07:58:53Z

@lee1043, I'm rebuilt distarray, cdms2 and cdutil (not yet available). Please try your command tomorrow and let me know what trouble you run into. It seems like @tomvothecoder was running into issues with pcmdi_metrics itself so please consider trying to fix above as well.

@xylar

Update setup.py following suggestion from @xylar at #893 (comment)

tomvothecoder · 2023-02-03T18:46:59Z

@lee1043, I'm rebuilt distarray, cdms2 and cdutil (not yet available). Please try your command tomorrow and let me know what trouble you run into. It seems like @tomvothecoder was running into issues with pcmdi_metrics itself so please consider trying to fix above as well.

@xylar I followed your instructions with release_version and it seems to have fixed the issue with installing pcmdi_metrics using python -m pip install (and python setup.py install).

I opened up PR #895 with the fixes and the console output. @lee1043 you are tagged for review.

lee1043 added the build label Feb 1, 2023

lee1043 added a commit that referenced this issue Feb 2, 2023

temporary solution for the setuptools issue (#893)

71b9282

xylar mentioned this issue Feb 3, 2023

Rebuild to fix strange version number from git hash conda-forge/cdutil-feedstock#20

Merged

4 tasks

lee1043 added a commit that referenced this issue Feb 3, 2023

Update setup.py

9047859

Update setup.py following suggestion from @xylar at #893 (comment)

lee1043 mentioned this issue Feb 3, 2023

Update setup.py #894

Closed

tomvothecoder mentioned this issue Feb 3, 2023

Replace setup.py versioning scheme with standard convention #895

Merged

lee1043 closed this as completed in #895 Feb 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

installation in development conda env fails #893

installation in development conda env fails #893

lee1043 commented Jan 31, 2023

lee1043 commented Feb 1, 2023

tomvothecoder commented Feb 2, 2023 •

edited

tomvothecoder commented Feb 2, 2023

xylar commented Feb 3, 2023

xylar commented Feb 3, 2023

xylar commented Feb 3, 2023 •

edited

xylar commented Feb 3, 2023

xylar commented Feb 3, 2023

lee1043 commented Feb 3, 2023

xylar commented Feb 3, 2023

xylar commented Feb 3, 2023

xylar commented Feb 3, 2023

xylar commented Feb 3, 2023

tomvothecoder commented Feb 3, 2023

installation in development conda env fails #893

installation in development conda env fails #893

Comments

lee1043 commented Jan 31, 2023

What happened:

To reproduce error:

1. Generate development conda env following this

2. From PMP local directory cloned from github

Error:

Temporary solution

1. Check setuptools version and downgrade setuptools

2. Check if working: Repeat python setup.py install that seems work for main branch

2-1. main branch

2-2. development branch

Options for resolving

lee1043 commented Feb 1, 2023

tomvothecoder commented Feb 2, 2023 • edited

tomvothecoder commented Feb 2, 2023

xylar commented Feb 3, 2023

xylar commented Feb 3, 2023

xylar commented Feb 3, 2023 • edited

xylar commented Feb 3, 2023

xylar commented Feb 3, 2023

lee1043 commented Feb 3, 2023

xylar commented Feb 3, 2023

xylar commented Feb 3, 2023

xylar commented Feb 3, 2023

xylar commented Feb 3, 2023

tomvothecoder commented Feb 3, 2023

1. Check `setuptools` version and downgrade setuptools

2. Check if working: Repeat `python setup.py install` that seems work for main branch

tomvothecoder commented Feb 2, 2023 •

edited

xylar commented Feb 3, 2023 •

edited