Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] build_scripts.py incorrectly handles shebang lines with DOS line endings #124

Closed
amalachow opened this issue Mar 4, 2022 · 4 comments · Fixed by #129 or pypa/setuptools#3224
Assignees

Comments

@amalachow
Copy link

Duplicating pypa/setuptools#3129 here due to some permissions issues during the transfer.

Python version

Python 3.8.10

OS

NT-10.0-19042

Description

shebang using DOS line endings causes the shebang regular expression to fail resulting in redirection to the python interpreter used for the installed script to not behave as expected.

Expected behavior

The shebang is rewritten consistently for UNIX and DOS line endings.

How to Reproduce

#!/bin/bash -eu
printf "[[System Info]]\n"
uname -srvmo
python3 --version
python3 -m virtualenv --version
rm -rf .venv
python3 -m virtualenv --quiet -p python3 .venv
.venv/Scripts/python -m pip install --quiet --upgrade pip &> /dev/null
.venv/Scripts/python -m pip install --quiet --upgrade setuptools &> /dev/null
.venv/Scripts/python -m pip list

rm -rf example
mkdir example
cat > example/example.py << END
#!/usr/bin/env python3

def main():
    print("Example script")

if __name__=="__main__":
    main()
END

cat > example/setup.py << END
from setuptools import setup
setup(name="example", scripts=["example.py"])
END

printf "\n[[Expected behavior]]\n"
file example/example.py
.venv/Scripts/pip install --quiet ./example
head -1 .venv/Scripts/example.py

printf "\n[[Issue]]\n"
sed -ie 's/\r*$/\r/g' example/example.py
file example/example.py
.venv/Scripts/pip install --quiet -I ./example
head -1 .venv/Scripts/example.py

fix="--- a/build_scripts.py
+++ b/build_scripts.py
@@ -15 +15 @@
-first_line_re = re.compile(b'^#!.*python[0-9.]*([ \t].*)?$')
+first_line_re = re.compile(b'^#!.*python[0-9.]*([ \t].*)?\r?$')"

printf "\n[[Proposed Fix]]\n"
echo "${fix}"
patch -u -p0 .venv/Lib/site-packages/setuptools/_distutils/command/build_scripts.py <<< ${fix}
file example/example.py
.venv/Scripts/pip install --quiet -I ./example
head -1 .venv/Scripts/example.py```

Output

[[System Info]]
MINGW64_NT-10.0-19042 3.1.7-340.x86_64 2021-03-26 22:17 UTC x86_64 Msys
Python 3.8.10
virtualenv 20.10.0 from C:\Python38\lib\site-packages\virtualenv\__init__.py
Package    Version
---------- -------
pip        22.0.3 
setuptools 60.9.3 
wheel      0.37.1 

[[Expected Behavior]]
example/example.py: Python script, ASCII text executable
#!C:\setuptools-issue\.venv\Scripts\python.exe

[[Issue]]
example/example.py: Python script, ASCII text executable, with CRLF line terminators
#!/usr/bin/env python3

[[Proposed Fix]]
--- a/build_scripts.py
+++ b/build_scripts.py
@@ -15 +15 @@
-first_line_re = re.compile(b'^#!.*python[0-9.]*([ \t].*)?$')   
+first_line_re = re.compile(b'^#!.*python[0-9.]*([ \t].*)?\r?$')
patching file .venv/Lib/site-packages/setuptools/_distutils/command/build_scripts.py
example/example.py: Python script, ASCII text executable, with CRLF line terminators
#!C:\setuptools-issue\.venv\Scripts\python.exe
@jaraco
Copy link
Member

jaraco commented Mar 12, 2022

I'm not certain the right thing to do here is support shell scripts containing carriage returns in the shebang. See this reference, which I found from Wikipedia on shebang. That answer suggests that it's incorrect or unsupported to encounter a carriage return in the shebang.

Before distutils enacts changes to support this additional, non-preferred form (versus the preferably one obvious way to do it), I'd like to see support from the Unix community that this format should be supported.

I'd also like to see a more thorough description of why it's important to support this format. Your repro above does a pretty good job of telling the story. In particular, it seems that if one uses simple unix tools (i.e. cat) to construct a script on a Windows machine, it will create the file containing a carriage return.

It's curious to me that Python's regex module detects "\n" as a new line, but does not detect "\r\n" as a new line. Perhaps that implies that distutils should have read the file using universal newlines in order to translate any \r\n combinations to \n. Regardless, I don't believe embedding \r in the regex is the right approach here. Instead, the input to first_line_re.match should have any trailing newline removed.

Also, I notice that when the shebang is replaced, it gets replaced without a carriage return. That seems bad. That will end up creating a script with mixed newlines (LF for the shebang, and CRLF for the rest), another reason to think carefully about whether it's worth supporting CRLF at all.

Finally, if we can determine that the best thing for Python/distutils to do here is support CRLF, I'd like to see some tests that capture this new requirement (i.e. test that copy_scripts works on a file created with CRLF in the shebang).

@amalachow
Copy link
Author

I'd also like to see a more thorough description of why it's important to support this format.

There are some behavioral implications of the shebang being rewritten inconsistently when using copy_scripts within a virutalenv. So if example.py uses a dependency installed in the virtualenv the shebang not being rewritten can result in running the script directly to fail as the shebang still points to the system interpreter.

I noted this failure condition as a result of inconsistency in the line endings ultimately stemming from git core.autocrlf auto vs input. Obviously this can be mitigated by being explicit about line endings in .gitattributes, but it seemed sensible that Unix and Windows could/should be handled consistently.

Outside the Unix expectations it looks like CPython's launcher supports "\r\n" which is worth considering given the nature of the reported issue as its predominantly Windows centric behavior that is being scrutinized.

I'm generally in agreement with your overall assessment on the mixed line endings as a side effect of the shebang rewrite and that simply including "\r\n" as consideration in the regex may not be the appropriate solution. I can work on getting some additional test cases added should this progress as something you consider worth rectifying.

@jaraco
Copy link
Member

jaraco commented Mar 15, 2022

Given the now near ubiquity of Linux on Windows, I think this is worth fixing. Here's what I'd like to see:

  • When reading a scripts file for copy, read it in using universal newlines. That implies also using text mode. Since the text could potentially contain multi-byte characters, use encoding='latin-1', which supports all bytes.
  • When writing a script file for copy, use the newline as detected by the read operation (i.e. retain newlines, i.e. outf = open(name, newlines=inf.newlines)).

This way, the style of newlines in the file is unchanged, but any processing of the file, such as replacing the shebang, can be done assuming \n for newlines and regex works naturally.

I'd definitely consider a patch with this approach. Let me know how it goes and if there are any assumptions I'm making that are broken.

@jaraco
Copy link
Member

jaraco commented Mar 27, 2022

  • Since the text could potentially contain multi-byte characters, use encoding='latin-1', which supports all bytes.

On second thought, use of utf-8 and the 'surrogateescape' handler would be better:

>>> raw = b''.join(val.to_bytes(1, 'big') for val in range(256))
>>> raw.decode('utf-8', errors='surrogateescape').encode('utf-8', errors='surrogateescape') == raw
True

@jaraco jaraco self-assigned this Mar 27, 2022
clrpackages pushed a commit to clearlinux-pkgs/pypi-setuptools that referenced this issue Jun 8, 2022
…version 62.3.2

Alex Hedges (1):
      Render distutils commit links properly in docs

Ananth Pattabiraman (3):
      add resources on packaging
      add 2674 changelog
      Update docs/userguide/quickstart.rst

Anderson Bravalheri (432):
      Add documentation about build_meta wrappers
      Add news fragment
      Mention environment markers as alternative
      Configure pytest to enable/disable integration tests
      Add integration test based on feedback from #2849
      Add tox environment for integration tests
      Run integration tests before release on CI
      Add news fragment about integration tests
      Disable pytest plugins in integration tests via -p no:*
      Use separated "extras" for integration tests
      Fix flake8 errors for conftest.py
      Simulate pip's isolation using virtualenv
      Use shutil to find executable
      Separate some reusable integration helpers
      Use '=' as setup.cfg delimiter for extras option
      Change vendoring script to preserve license files
      Add news fragment
      Remove license lookup in *.egg-info
      Improve path handling on move_license task
      Remove workaround for test/CI dependency on Sphinx
      Rename `docs/{images/README => artwork}.rst`
      Replace in-tree sphinx extension with sphinx-favicon
      Use an SVG optimiser
      Add news fragment
      Remove unused imports from docs/conf.py
      Add SETUPTOOLS_USE_DISTUTILS configuration to integration tests
      Allow integration tests to be manually triggered in GitHub Actions
      Add fixtures for sdist and wheel artifacts
      Extract venv fixtures from test_distutils_adoption
      Replace tmp_src fixture with the virtualenv fixtures
      Remove dependency on pytest_virtualenv
      Prevent some tests from inadvertently using the project root for builds
      Add news fragment
      Use setuptools wheel instead of source tree in integration tests
      Ensure test_cygwin pass before CI release action
      Fix misplaced news fragment
      Remove `numpy` sdists from integration tests
      Add test to make sure 3.10 is not interpreted as 3.1
      Prevent test files from being included in setuptools own wheel
      Fix failing test when user site-packages has no version number
      Remove deprecated communication platforms
      Add test to ensure the correct log level is set
      Fix weird distutils.log reloading/caching situation
      Avoid replacing the global log level in test
      Update setuptools/logging.py
      Add tests about duplicated distutils imports
      Use distutils.cmd to test duplicated imports
      Fix codecov config
      Move lingering 'docs' files into correct sections in changelog
      Modify tools.finalize.check_changes to validate extensions
      Ignore flake8/black warnings with pytest 7.0.1 (jaraco/skeleton#58)
      Add timeout to test_build_meta
      Prevent type error from env var
      Kill process pool after timeout in test_build_meta
      Kill individual worker process after timeout in test_build_meta
      Ensure process killing does not fail on Windows
      Reduce GitHub Actions timeout from 6h to 75min
      Disable coverage on PyPy
      Skip pytest-cov in setup.cfg for PyPy
      Prevent CI from trying to use '--cov' when pytest-cov is not installed
      Treat codecov action as informational only
      Prevent codecov from polluting the diff view in PRs
      Tweak concurrency group for CI so tags are not cancelled by pushes
      Attempt to workaround yet another problem with PyPy tests
      Add news fragment
      Avoid accidental URL replacement in CHANGELOG
      Extend matching of issue/commit links to all PyPA repositories
      Fix some URLs that are being incorrectly displayed
      Add news fragment
      Improve regex for autolinking PEPs on changelog
      XFAIL on OSError in test_build_meta for PyPY
      Don't warn on false positive for author/maintainer's email
      Check for file existence before using samefile
      Just check for if destination file exists
      Apply suggestions from code review
      Add links to MANIFEST.in docs and clarify data files inclusion
      Add a note about data files being read-only
      Modify datafiles docs to emphasize importlib.resources over pkg_resorueces
      Add section about distributed files to miscellaneous
      Add link to MANIFEST.in in quickstart
      Add news fragment
      Add missing link to PyPUG MANIFEST.in docs
      Add another missing link to PyPUG
      Avoid using a same set of words repeatedly too close
      Emphasize data files should be included inside the package
      Apply suggestions from code review
      Clarify the relationship between wheel <> sdist
      Apply suggestions from code review
      Update docs/userguide/miscellaneous.rst
      Move *PackageFinder to the new 'discovery' module
      Add a more careful package finder for flat-layout use case
      Add module finder
      Refactor finders to share code via common base
      Cleanup test imports
      Add tests to specify automatic option discovery
      Add implementation for automatic config discovery
      Add news fragment
      Add build as a test dependency
      Exclude subpackages in FlatLayoutPackageFinder
      Fix rst markup on news fragments
      Fix path handling on Windows
      Add documentation about automatic discovery
      Attempt to improve autodoc
      Improve news fragment
      Small doc improvement for package_discovery
      Improve autodoc for finder exclude
      Improve text in package discovery docs
      Add examples for package_dir customisation
      Sync setup.cfg/.py examples for package_dir
      Small fixes for text in package_discovery
      Exclude 'bin' dir in discovery
      Add news fragment with instructions for empty distributions
      Remove repeated dependency
      Ignore some other folders and files by default
      Reuse integration helper
      Avoid importing a test inside other test
      Ignore build and dist folders in flat-layout
      Improve news fragment about breaking change
      Don't overwrite if the user specifies empty packages/py_modules
      Test auto-discovery with explicit variation of src layout
      Rename `config` to `config.setupcfg`
      Extract post-processing functions from config
      Allow root_dir to be explicit in config.expand functions
      Allow single strings in config.expand.read_files
      Adequate test_setupcfg to latest changes in setupcfg
      Add news fragment
      Make __all__ immutable in setuptools.config
      Split complex generator expression in setuptools.config.expand
      Adopt review suggestions
      Replace pushd with monkeypatch.chdir in test_expand
      Ensure proper exception matching in test_expand
      Parametrize test_expand.test_find_packages
      Add `tomli` as vendorised dependency
      Add `validate-pyproject` as a vendored dependency
      Add news fragment
      Make comment in setuptools/_vendor/vendored.txt more clear
      Ensure relative imports for vendorised tomli
      Update vendored tomli to 2.0.1
      Improve custom vendoring logic for validate-pyproject
      Update vendored validate-pyproject to 0.4
      Update vendored validate-pyproject to 0.5.2
      Implement read_configuration from pyproject.toml
      Expand dynamic entry_points from pyproject.toml
      Make include_package_data=True for `pyproject.toml` configs
      Add means of applying config read from pyproject.toml to dist
      Add the apply_configuration API to setuptools.config.setupcfg
      Test pyproject.toml config has the same effect as setup.cfg
      Fix pyproject config when tool table is not present
      Remove no longer needed tomli import workaround
      Update version of test dependency 'ini2toml'
      Prevent resource warnings in test_apply_pyprojecttoml
      Add a 'uses_network' marker to tests that require connectivity
      Update test dependency ini2toml to 0.8
      Avoid failing due to 3rd party config in pyproject.toml
      Add tests against "empty" pyproject.toml
      Avoid using pkg_resources for entry points
      Back-fill Description-Content-Type according to readme suffix
      Update test dependency ini2toml to 0.9
      Add pyproject.toml to dist.parse_config_files
      Add deprecation notice for config.{read,parse}_configuration
      Add backend test with pyproject.toml-based configs
      Test editable installs with pyproject.toml metadata
      Replace skip in editable install test with xfail
      Add news fragment
      Ensure build_meta don't have problems with instructions after setup()
      Test if not-zip-safe file is being generated with project metadata
      Test static metadata in pyproject.toml is not overwritten by setup.py
      Explicitly inform users that pyproject.toml config is experimental
      Rely on validate-pyproject default errors
      Show significant error messages to user and avoid traceback pollution
      Removed unused import
      Fix variable name error
      Add some type hints to config.setupcfg
      Add some type hints to config.pyprojecttoml
      Add some type hints to config.expand
      Find namespaces by default when using config in 'pyproject.toml'
      Add news fragment
      [CI] Allow pre-built distribution to be used in tests with virtualenvs
      Add venv to the default exclude list
      [CI] Disable test_pip_upgrade_from_source when network if off
      Update changelog.d/3147.misc.rst
      Replace direct usage of the `py` library
      Fix wrong order when partitioning TOML config files
      Import package finders directly from discovery module
      Add test capturing the expectation of package_dir being autodiscovered
      Allow package_dir autodiscovery for setup.cfg
      Externalize find_parent_package from discovery class
      Allow expand.find_packges to fill package_dir
      Make sure package_dir is populated before processing cmdclass and 'attr:' in setup.cfg
      Postpone expanding dynamic config in pyproject.toml
      Capture expectations about discovery and attr/cmdclass
      Take discovery into account when expanding pyproject.toml
      Mark features related auto-discovery and pyproject metadata as experimental
      Add news fragment
      Fix small errors in docs
      Test the behavior of license and license-files in pyproject.toml
      Update vendored dependency validate-pyproject to 0.6
      Change pyproject.toml tests to not use dynamic for license/license-files
      Change pyproject.toml processing to not use dynamic for license/license-files
      Use windows-2019 for tests in GitHub Actions
      Fix problem with path objects for Windows
      Attempt to re-enable Windows tests
      Use windows-2019 for tests in GitHub Actions
      [Docs] Improve documentation about migration from distutils
      Fix version of setuptools for default local distutils
      Fix PEP 632 link display
      Link packaging
      Exclude PyPy+Windows from test matrix
      Just skip the most problematic test for PyPy on Windows
      Update setuptools/tests/test_wheel.py
      Extract reusable _unpack_zipfile_obj from archive_utils
      Use function from archive_util instead of overwritting ZipFile
      Run the test on Windows, but don't check the file mode
      Fix towncrier command in tools/finalize
      Bump version: 60.9.3 → 60.10.0
      Improve package discovery docs
      Add initial docs about pyproject.toml metadata
      Add news fragment
      Add remark about editable installs
      Mention experimental pyproject.toml support in discovery docs
      Mention experimental pyproject config in the quickstart
      Add some missing references in the quickstart
      Fix link to keywords
      Clarify deprecated fields for tool.setuptools
      Improve discovery section in the quickstart
      Improve entry-points section in the quickstart
      Improve dependencies section in quickstart
      Improve data files section in quickstart
      Add note about editable installs in quickstart
      Clarify editable installs note in quickstart
      Improve notes on quickstart
      Fix references to layouts in docs
      Add tab for pyproject.toml in dependency management docs
      Clarify extras in entry-points are deprecated
      Add notes to pyproject_config docs
      Apply suggestions from code review
      Attempt to re-enable Windows tests
      Small doc improvements
      Clarify directives in the context of pyproject.toml
      Add tests for dynamic classifiers on pyproject.toml
      Split lines for dynamic classifiers in pyproject.toml
      Add expectations about multiple packages for discovery
      Add other names to the list of excluded packages for auto-discovery
      Allow type stubs for FlatLayoutPackageFinder
      Refactor function for finding top-level packages in auto-discovery
      Add function to remove stubs from a list of packages
      Remove stubs when trying name auto-discovery
      Avoid running build unless necessary in test for discovery
      Prevent accidental multi-package dist with auto-discovery
      Adequate existing tests to the new errors for auto-discovery
      Improve logs/docstrings for setuptools.discovery
      Add type hints to setuptools.discovery
      Fix type error in setuptools.config
      Update package discovery docs to reflect latest changes
      Fix error in doctest on Windows
      Change tabs in discovery docs to be similar to quickstart
      Fix problem caused by mispelling of py_modules for pyproject.toml
      Prevent setup_requires patches from activating auto-discovery
      Simplify package name condition for flat layout
      Use the same comment as distutils
      Attempt to clarify which url is missing for pyproject-metadata builds
      Use blank lines to emphasize warnings
      Refactor ConfigDiscovery._root_dir as a property
      Fix test missing assertion
      Improve interaction between pyproject.toml metadata and discovery
      Rename variable alias
      Ensure empty package_dir is not replaced on auto-discovery
      Make sure to ignore option errors with MinimalDistribution
      Test discovery when ext_modules are provided
      Skip discover when ext_modules are provided
      Update discovery docs to mention ext_modules
      Ignore ext-modules for auto-discovery with pyproject.toml metadata
      Adequate docs to the latest changes
      Improve organisation of test_config_discovery
      Add a few other reserved package/module names to discovery
      Add test for default include-package-data with 'pyproject.toml'
      Fix default include-package-data with 'pyproject.toml'
      Use better variable naming
      Add news fragment
      Restore tip about editable installs
      Fix example of environment maker for dependencies in quickstart
      Improve text about CLI entry-point in quickstart
      Add module to entry_point example in quickstart
      Add missing preposition
      Improve note about setup.py
      Remove note about setup.py being required for editable installs
      Clarify that only deprecated fields should be avoided in pyproject_config
      Specify Python 3 after Python 2 in intersphinx mapping
      Avoid extlink for issue to prevent verbose warnings
      Fix wrong version reference in quickstart
      Fix edge case of package discovery
      Add the upcomming toxfile.py to the list of ignored modules for flat-layout
      Specify that some builds may break due to improper configuration
      Add back notes about editable install and pyproject metadata
      Add unit test for read_attr
      Change tests for resolve_class to consider different layouts
      Bump version: 60.10.0 → 61.0.0
      Avoid unnecessarily changing package_dir
      Add news fragment
      Attempt to solve pathsep problems in windows tests
      Add missing breaking change note
      Test setup.py' include_package_data not ignored when parsing pyproject
      Avoid overwritting dist.include_package_data with default
      Add news fragment
      Add back convert_path as deprecated function
      Add news fragment
      Make sure dynamic classifiers don't fail on unexisting files
      Add news fragment
      Disable auto-discovery when the 'configuration' attribute is passed
      Update news fragment
      Bump version: 61.0.0 → 61.1.0
      Add test for setup.py install and dependencies
      Make install consider dist.run_command is overwritten in v61.0.0
      Add news fragment
      Add workaround for PyPy
      Temporarily disable test for Windows+PyPy
      Add debug statements for test on Windows
      XFAIL test due to uncorrelated reason
      Fix test for setup.py
      Improve news fragment
      Bump version: 61.1.0 → 61.1.1
      Test popular invalid pyproject patterns
      Temporarily forgive popular patterns on invalid pyproject.toml
      Add news fragment
      Add missing re-raise statement
      Update pyproject validation as generated by validate-pyproject==0.6.1
      Remove no longer necessary xfail mark
      Remove left-over debug statement
      Add news fragment
      Improve news fragment
      Tests mixed pyproject metadata + config from setup.py
      Warn if a project metadata is set outside of pyproject without dynamic
      Restructure config.pyproject to consider "pre-set" dynamic values
      Adequate existing tests for the latest changes
      Consider missing edge case for tool.setuptools.dynamic in pyproject
      Improve error/warning messages
      Prepare to be strict in the future about entry-points in pyproject
      Add news fragment
      Store install_requires and extras_require for future usage
      Make sure apply function remains private
      Ensure pyproject.toml does not break dynamic install_requires
      Merge pre-set dependencies when applying pyproject
      Small refactor
      Preserve _tmp_extras_require as an ordered set
      Decrease verbosity of _install_setup_requires
      Add news fragment
      Fix incorrect PR number
      Fix previous detection of empty arrays
      Minor change on news fragments
      Fix invalid link on changelog
      Bump version: 61.1.1 → 61.2.0
      Test dist_info creates similar dir to bdist_wheel
      Capture expectation of invalid version warning
      Fix duplicated version tags in egg_info
      Restore tags in egg_info but change the idempotency check
      Change dist_info naming to use the same convention as bdist_wheel
      Add comment explaining test
      Add news fragment
      Separate vendoring script and code generator for pyproject validations
      Move _validate_pyproject to config
      Ignore coverage in automatically generated code
      Update version of validate-pyproject
      Fix flake8 errors
      Fix unintentional mistake in config/pyproject
      Update validate-pyproject to v0.7.1
      Disable automatic download of trove classifiers by default
      Add news fragment
      Update .coveragerc
      Test with different package names
      Fix reference to issue number
      Bump version: 61.2.0 → 61.3.0
      Add metatest to make sure auxiliary file is properly packaged
      Fix missing file in manifest
      Split download helpers to their own file
      Add script that allow users to preload examples for offline testing
      Add news fragment
      Fix flake8 problems
      Add comments with instructions for developers
      Bump version: 61.3.0 → 61.3.1
      Fix error with test_easy_install
      Be explicit about packages
      Attempt to fix problems on windows
      Add test for dynamic readme from setup.py args
      Fix dynamic readme
      Add news fragment
      Rename news fragment file
      Bump version: 61.3.1 → 62.0.0
      Simplify auto-discovered package_dir
      Add news fragment
      Rename variable
      Update version of pyparsing in vendored.txt
      Update pyparsing to fix problems with 3.11
      Add news fragment
      Cache downloaded files used during tests for setuptools.config
      Fix YAML error
      Try to rescue the download backing off a few seconds
      Add news fragment
      Warn when an incompatible version of importlib is used
      Add news fragment
      Add warning about pathlib to changelog
      Prefer `importlib-metadata` in warning message
      Fix incorrect information in comment
      Capture expectations about overwritten dependencies in test
      Add warning about overwritten dependencies
      Add news fragment
      Add test that capture transitional behaviour for build_py
      Warn about packages/modules included as package data
      Add news fragment
      Quickfix missing comment mark in docs
      Simplify checks for abuse of include_package_data
      Avoid mentioning that pip is used for download
      Add markup to highlight deprecation of test-related keywords
      Fix markup on keywords documentation
      Add introduction text to keywords docs
      Add anchors for keywords in docs.
      Make the separation between distutils and setuptools keywords more clear
      Move userguide/keywords to deprecated/changed_keyworkds
      Update docs/deprecated/changed_keywords.rst
      Apply suggestions from code review
      Add deprecation messages for `namespace_packages`.
      Add news fragment
      Add deprecation notices to docs
      Bump version: 62.2.0 → 62.3.0
      Add news fragment
      Bump version: 62.3.0 → 62.3.1
      Improve warning message for _IncludePackageDataAbuse
      Improve warning message for _WouldIgnoreField
      Improve warning message for dist_info._version
      Improve warning message for _InvalidFile
      Add news fragment
      Update setuptools/command/build_py.py
      Bump version: 62.3.1 → 62.3.2

Andrew Murray (3):
      Fixed typos [ci skip]
      Only import ctypes when necessary
      Fixed typo

Andrey Bienkowski (6):
      XXX: Debugging
      Test: editable install \w --user&build isolation
      Revert "XXX: Debugging"
      Fix editable --user installs with build isolation
      Retry CI
      Fix test_editable_user_and_build_isolation

Anthony Sottile (2):
      Fix AttributeError when sysconfig does not supply platsubdir
      set origin= for distutils.__spec__

Binjian (1):
      Update package_discovery.rst

Christian Heimes (7):
      Speedup startup of Python by importing less
      Add changelog entry
      Temporarily add back filter, tests fail without it
      Use internal warnings API and _TrivialRe hack to install filter
      Move filter into meta finder
      Suppress distutils deprecation warning
      pytest.ini uses tabs

Christoph Reiter (14):
      msvc9compiler: Don't raise DistutilsPlatformError on import
      tests: skip uid/gid using tests under Cygwin
      CI: add a CI job for testing under Cygwin
      cygwinccompiler: Split CC env var before passing to subprocess
      sysconfig: use get_config_h_filename() from the stdlib
      sysconfig: use parse_config_h() from stdlib sysconfig
      tests: use loadTestsFromTestCase() instead of the deprecated makeSuite()
      tests: fix usage of requires_zlib() decorator
      tests: fix tests on Ubuntu 22.04
      tests: use sys.executable instead of hardcoding "python"
      sysconfig: use get_makefile_filename() from stdlib sysconfig
      util.get_host_platform: restore Python 3.9 behaviour for Python 3.7/3.8
      CI/cygwin: use cygwin/cygwin-install-action instead of choco
      CI: re-enable the cygwin jobs again

Chuck McCallum (1):
      Small wording tweaks for readability

Dominic Davis-Foster (1):
      Skip non-string values from sysconfig.get_config_vars()

FX Coudert (1):
      bpo-42504: fix for MACOSX_DEPLOYMENT_TARGET=11 (GH-23556)

Giulio Procopio (1):
      [maint] removed unneded exception inclusion

Gyeongjae Choi (1):
      Remove comma

Isuru Fernando (12):
      Use mock.patch.dict to patch os.environ
      Use sysconfig to provide get_config_vars
      When building C++ extensions, replace all of linker command
      add a test for CC with two words
      Update the test description
      Add some comments
      Update test to check for correct value
      Fix test
      Pass through PROGRAMDATA, PROGRAMFILES env variables
      Re-enable windows tests
      Fix EXT_SUFFIX for windows py<3.8
      Fix SO too

Jason R. Coombs (254):
      Restore local distutils as the default.
      Update changelog.
      Consider this a breaking change.
      Extract 'incomplete_scheme' as a variable for readability.
      Extract _pypy_hack method.
      Honor sysconfig.get_preferred_scheme when selecting install schemes. Fixes #76.
      Extract grp/pwd handling to a unix_compat module.
      Remove unused imports in cygwincompiler and the reliance on them in tests.
      Add changelog
      Bump version: 59.7.0 → 59.8.0
      Bump version: 59.8.0 → 60.0.0
      Extract _select_scheme function to be re-used by Setuptools.
      In easy_install, re-use scheme selection from distutils if available.
      Add fallback support for distutils in stdlib.
      Update changelog.
      Bump version: 60.0.0 → 60.0.1
      Select 'posix_user' for the scheme unless falling back to stdlib, then use 'unix_user'. Fixes #2938.
      Bump version: 60.0.1 → 60.0.2
      Extract frame_file_is_setup.
      Avoid KeyError in distutils hack when pip is imported during ensurepip. Fixes #2940.
      Bump version: 60.0.2 → 60.0.3
      Add support for 'platsubdir'. Fixes pypa/distutils#85.
      Update changelog.
      Bump version: 60.0.3 → 60.0.4
      When headers are missing in a preferred scheme, use the default scheme as a fallback to resolve headers. Workaround for and fixes #88.
      Cast value to str and provide a comment explaining why. Ref pypa/distutils#87.
      Update changelog
      Move 'str' cast into the one place where it's needed and allow config_vars to have None values.
      Bump version: 60.0.4 → 60.0.5
      In distutils_hack, only add the metadata finder once. In ensure_local_distutils, rely on a context manager for reliable manipulation. Fixes #2958.
      Extend tests to capture expectation of only one DistutilsMetaFinder at a time.
      Update changelog.
      Bump version: 60.0.5 → 60.1.0
      Add discord badge
      Include channel in the URL.
      👹 Feed the hobgoblins (delint).
      Remove skipif for Python 3.6, no longer supported.
      Mark test as xfail. Fixes #2975.
      Bypass distutils loader when setuptools module is no longer available on sys.path. Fixes #2980.
      Bump version: 60.1.0 → 60.1.1
      Use line-based matrix values for nicer diffs. Remove Python 3.6 and bump to Python 3.10, matching jaraco/skeleton and pypa/setuptools approaches.
      Leave sphinx unpinned as found upstream.
      Add setuptools.log to supersede distutils.log. Ref #2973.
      Monkey patch distutils.log.set_threshold so the Python logger honors calls to it.
      Restore assertion about expected distutils.
      Check early for the presence of local distutils and bail out if the found version of Setuptools doesn't have distutils.
      Update changelog.
      Restore 'add_shim' as the way to invoke the hook. Avoids compatibility issues between different versions of Setuptools with the distutils local implementation. Renamed the former 'add_shim' as 'insert_shim'. Fixes #2983
      Restore 'get_versions' attribute, allowing older mpi4py to monkeypatch it. Fixes pypa/setuptools#2969.
      Disable setuptools installation. Fixes pypa/distutils#99.
      Also use PEP 517 to build things like pytest-virtualenv.
      39 is actually required to get the right packages.
      Unset VIRTUALENV_NO_SETUPTOOLS for ci_setuptools because pytest-virtualenv can't install without it. Ref man-group/pytest-plugins#190
      It really must be literally 39.
      Add changelog.
      Bump version: 60.1.1 → 60.2.0
      In tests, add compatibility shim py38compat.requires_zlib.
      Include Python 3.8 in tests. Ref pypa/distutils#100.
      Add another exception for the exclusion for pip. Fixes #2993.
      Add DictStack from jaraco.collections.
      In install command, honor config_vars from sysconfig. Fixes pypa/distutils#103.
      Use 'dict()' instead of '.copy()', for compatibility with DictStack.
      Update changelog.
      Bump version: 60.2.0 → 60.3.0
      Update DictStack implementation from jaraco.collections 3.5.1
      Update changelog.
      Add test capturing failure. Ref #3002.
      Suppress AttributeError when detecting get-pip. Fixes #3002.
      Remove the env, as the test suite runs local by default but also tests stdlib.
      Extract the SYSTEMROOT handler and document it.
      Bump version: 60.3.0 → 60.3.1
      Capture performance of interpreter startup. Ref #3006.
      Check that distutils has an origin. Ref #2990.
      Bump version: 60.3.1 → 60.4.0
      Bump version: 60.4.0 → 60.5.0
      Disable cygwin tests for now. Ref #3016.
      Instead of detecting 'get-pip' during the import of pip, detect the attempt to 'import setuptools' during 'get-pip', and in that case, stub the import to signal the presence of setuptools. Ref #3022. Fixes #2993.
      Update changelog.
      Bump version: 60.5.0 → 60.5.1
      Bump version: 60.5.1 → 60.5.2
      Trim excess indentation.
      Honor sysconfig variables in easy_install. Fixes #3026.
      Bump version: 60.5.2 → 60.5.3
      Create tox routine for updating vendored packages.
      Move vendor routine to tools
      Remove dependency on paver in vendored update.
      Update vendored with no changes.
      six isn't installed any longer
      Simplify vendored script to simply include the metadata.
      Move towncrier template to tools
      Move launcher build scripts to tools
      Remove filtering of distutils warnings. Ref #3009.
      Refactor to generalize script detection.
      Suppress distutils replacement when building or testing CPython. Fixes #2965. Fixes #3007.
      Bump version: 60.5.3 → 60.5.4
      Update documentation to match and remove requirements file with stale references.
      Remove noise in the changelog.
      Rely on pip_run.launch to install sitecustomize.
      Replace IRC link with Discord. Restore default for blank issues.
      Use the generic invite link to pypa discord. Has better redirect behavior.
      Remove CoC checkbox as unnecessary friction.
      Add tests for yield_lines.
      Extract function for dropping comments.
      Extract function for joining continuations in lines.
      Consolidate behavior now that it fits on one line.
      Only validate name
      Add test capturing expectation when name is not supplied.
      Remove hook in run_commands; a different hook point may be better.
      Refactor to calculate missing as a difference of sets. Simply emit all missing keys as repr() of that difference.
      Remove name from distributions where the name is irrelevant to the tests.
      Only resolve 'parent' directory when xdist is in use. Fixes #3059.
      Vendor jaraco.text to supply yield_lines, drop_comment, and join_continuation.
      Include all the dependencies needed to run on Python 3.7 and patch all of them to work in a vendored environment.
      Ensure text file from vendored package is included.
      Remove workaround to add __init__ for vendored jaraco package, seemingly unnecessary.
      Refactor to limit indentation and share behavior.
      Only rely on py_version_nodot_plat where not present (Python 3.9 and earlier).
      Add changelog
      Add py_version_nodot_plat substitution support to easy_install.
      Bump version: 60.5.4 → 60.6.0
      Update badge year
      Remove setup.py, no longer needed.
      Update changelog.
      Update changelog.
      Create a function for only_strs to help document its purpose.
      Bump version: 60.6.0 → 60.7.0
      Remove 'lorem_ipsum' property from jaraco.text, bypassing the behavior on import and other issues.
      Bump version: 60.7.0 → 60.7.1
      Add exclusions for pytest 7 deprecations in plugins. Fixes jaraco/skeleton#57.
      Add importlib_resources and importlib_metadata to vendored packages.
      Add module for selectively loading importlib modules.
      Move ensure_directory into setuptools.
      Refresh importlib_metadata to 4.10.1.
      Vendor jaraco.text with setuptools.
      In build_meta, remove dependency on pkg_resources.
      Pin vendored importlib dependencies for consistency.
      Update changelog.
      Move requirements processing to _reqs module. Add parse function.
      Replace use of parse_requirements with simple constructor.
      Replace use of parse_requirements with _reqs.parse.
      Update changelog.
      Use the parent category PytestDeprecationWarning, which is available on older pytest versions. Fixes jaraco/skeleton#57.
      Bump version: 60.7.1 → 60.8.0
      Ensure that _vendor/jaraco is available as a module. Fixes #3084.
      Bump version: 60.8.0 → 60.8.1
      Replaced use of iter_entry_points in setuptools.dist
      Update vendoring for importlib_metadata to support vendored dependencies typing_extensions and zipp.
      Migrate remainder of 'iter_entry_points' to importlib_metadata.
      Avoid dual-use variable.
      Remove duplicate check on ep is None.
      Refactor to construct data in a single expression and extract 'to_str'.
      Refactor to extract entry_points_definition generation.
      Extract function for converting entry points to a string.
      In egg_info, port use of pkg_resources.EntryPoint to importlib.metadata
      In test command, rely on metadata.EntryPoint for loading the value.
      Port check_importable to metadata.EntryPoint
      Extract module for entry point management.
      Prefer jaraco.text for yield_lines.
      Refactor _entry_points to separate loading from rendering. Explicitly validate and restore validation of entry points that don't match the pattern.
      Use new _entry_points.load to validate entry points.
      Update test_sdist not to rely on pkg_resources.
      Update changelog.
      Prefer jaraco.text for yield_lines.
      Bump version: 60.8.1 → 60.8.2
      Removed bootstrap.py script, no longer needed.
      Bump pytest-mypy and remove workaround for realpython/pytest-mypy#131.
      Require jaraco.packaging 9 adding compatibility for projects with no setup.py file.
      Disable running of sage-ci tests except on tagged commits. Ref #3093.
      Remove workaround for pypa/get-pip#137.
      Bump packaging to 21.3
      Use always_iterable to fix --global-option in one expression.
      Remove duplicate invocations of _BuildMetaBackend._fix_config.
      Remove new style class declaration, now the default.
      Make _BuildMetaBackend._fix_config static and add tests.
      Update changelog
      Update changelog.
      Re-use ensure_valid in validate.
      Bump version: 60.8.2 → 60.9.0
      Remove invocation of bootstrap script, no longer needed. Fixes #3100.
      Limit tests for stdlib distutils to one job. Ref #3093.
      Remove invocation of bootstrap script in release process.
      Include mention of bootstrap script removal.
      Add test for loading entry points from a string. Ref #3103.
      Fix issue where string-based entry points would be omitted. Fixes #3103.
      Prevent vendored importlib_metadata from loading distributions from older importlib_metadata. Fixes #3102.
      Bump importlib_metadata to 4.11.1. Fixes #3107.
      Bump version: 60.9.0 → 60.9.1
      When loading distutils from the vendored copy, rewrite __name__ to ensure consistent importing from inside and out. Fixes #3035.
      Update changelog
      Bump version: 60.9.1 → 60.9.2
      Add changelog
      Bump version: 60.9.2 → 60.9.3
      Deprecated upload_docs command. Ref #2971
      Remove tests for upload_docs. Removes test dependency on sphinx. Ref #2971.
      Prefer range().__contains__ for bounds check.
      Use samefile from stdlib, supported on Windows since Python 3.2.
      Extract method to validate optimize parameter.
      Remove 'isinstance(int)' check and just validate unconditionally.
      Move normpath into if block.
      Extract method for processing site dirs
      Disable tests on Windows while build issues exist. Ref pypa/distutils#118.
      Disable tests on Windows while build issues exist. Ref pypa/distutils#118.
      Use super throughout.
      Refactor UnixCCompiler.link, extracting two functions and reducing mccabe complexity from 15 to 11.
      Extract darwin detection logic to macos_compat module and do it once.
      Vendor nspektr. Utilize it in Distribution._install_dependencies.
      Update changelog.
      👹 Feed the hobgoblins (delint).
      Remove reference to non-existent doc.
      Fix warnings for pypi references.
      Fix docs build errors in changelog.
      Extract _linker_params function to capture the concerns about matching the compiler.
      Reword comment and refactor logic to add context and use imperative voice.
      Prefer docstrings to describe test intentions.
      👹 Feed the hobgoblins (delint).
      Add upstream reference.
      Update changelog
      👹 Feed the hobgoblins (delint).
      Rewrite docstring for imperative voice and proper structure.
      Move 'updated_files' operation outside of if statement as it appears in both branches unconditionally.
      Extract method for copying a file.
      Remove outfiles, unused.
      Extract _change_modes and _change_mode functions.
      Rewrite the comment to match the implementation.
      Use 'shebang_' for pattern and match.
      Extract method to validate the shebang.
      Restore Setuptools compatibility.
      In build_scripts, open scripts as text. Fixes pypa/distutils#124.
      Update changelog
      Move EXT_SUFFIX support to _py39compat, to be removed after support for Python 3.9 is dropped. Fall back to default behavior on Python 3.10. Remove functionality for SO, which has been long deprecated and is untested.
      👹 Feed the hobgoblins (delint).
      Just modify the vars in place.
      Restore expectation that SO matches EXT_SUFFIX with rationale.
      Get the version logic correct.
      Move compatibility concerns out of the function to do the adding.
      Emit warning after parsing. Fixes pypa/distutils#122.
      Disable installation of Setuptools in tox instead of GHA. Ref pypa/distutils#99.
      Refactor as simple replace. If a full string substitution proves to be necessary, let's create a mapping.
      Update changelog.
      Bump version: 62.0.0 → 62.1.0
      Extract check for 'd is None'.
      Combine logic between prefix and base_prefix.
      Adapt style
      Simply emit the expected value instead of converting to an exit code and deciphering.
      Rely on jaraco.envs for environment creation.
      Prefer tabs
      Use pass_none decorator.
      Prefer tox 3.25 with built-in support for Windows env vars.
      Run tests using tox, even on cygwin.
      Prefer tabs
      Update changelog. Ref #3299.
      Update changelog
      Try Python 3.9
      Bump version: 62.1.0 → 62.2.0

Jeremy Paige (4):
      Sync references keywords doc from userguide
      redirect userguide keywords to references keywords
      add changelog
      update changelog to cover new chagnes

Jeroen Ruigrok van der Werven (1):
      Replace the defunct distutils-sig ml with forum pointer

Jim Garrison (1):
      Clarify that `pyproject.toml` config will not be removed

Josip Delic (4):
      Add test for zipefile mode
      Fix ZipFile mode not set
      Deactivate tests on windows
      Add in changelog.d a note

Julien Palard (1):
      doc: Fix trailing spaces, tabs, and missing newlines at end of file.

Karolina Surma (2):
      Prevent leaking PYTHONPATH to spawned processes in tests
      Add news fragment

Maciej Pasternacki (6):
      Remove more_itertools.more from vendored libs (fixes pypa/setuptools#3090)
      Clean also .pyi of more_itertools
      Add changelog
      Restore accidentally deleted .coveragerc
      Apply suggestions from code review
      Restore more_itertools.more, make importing `concurrent.futures` lazy

Marc Mueller (1):
      Don't include Home-page with UNKNOWN value

Markus Bong (1):
      fix typo

Mathieu Kniewallner (1):
      Fix `bellow` typo in docs

Matt Davis (2):
      fix failures w/ py3-only loaders
      distutils shim should ignore setuptools on another path

Matt Hall (1):
      Some small typos and grammar fixes

Matthew Suozzo (6):
      Maintain `requires` order in METADATA.
      Revert inner function name change.
      Add news entry.
      Remove pkg_resources nondeterminism.
      Add new test to document expected order stability.
      Use MappingProxyType to ensure immutability.

Matthias Koeppe (6):
      .github/workflows/main.yml: Add test_cygwin
      .github/workflows/ci-sage.yml: New
      .github/workflows/ci-sage.yml: Remove flaky platforms, add new platforms
      .github/workflows/ci-sage.yml: Remove fedora-36, not ready
      .github/workflows/ci-sage.yml: Use https://trac.sagemath.org/ticket/33288 for CI fixes
      .github/workflows/ci-sage.yml: Replace centos-8 (EOL) by centos-stream-8, centos-stream-9

Michał Górny (3):
      Use cache_tag in default build_platlib dir
      Skip test_get_makefile_filename on non-CPython
      Update test_home_installation_scheme for pypy install paths

Naveen M K (3):
      MinGW/Cygwin: Remove checks for ancient gcc/binutils
      Raise a `DeprecationWarning` for `gcc_version` attribute
      util.get_host_platform: use sysconfig.get_platform

Pablo Cárdenas (1):
      Fix typo in quickstart section

Paul Ganssle (1):
      WIP: Reject packages without required metadata

Ronald Oussoren (2):
      bpo-41100: Support macOS 11 and Apple Silicon (GH-22855)
      bpo-42504: Ensure that get_config_var('MACOSX_DEPLOYMENT_TARGET') is a string (GH-24341)

Steve Dower (6):
      Fixes #112 install command doesn't use platform in nt_user scheme
      Fix fake expanduser() and verify calculated headers path
      Add non-NT path for expected path
      Change API
      Special case for Windows
      Detect when a venv is created from an in-tree build. Fixes #132

Steven Silvester (3):
      Update main.yml
      Update .github/workflows/main.yml
      Add concurrency limit to CI

Sviatoslav Sydorenko (2):
      Stop mentioning `wheel` in the context of PEP 517
      Add a change note for PR #3056

Vladimir Berlev (1):
      Fix typo in docs

Xing Han Lu (2):
      Update entry_point.rst
      Add news fragment

eacheson (1):
      Update quickstart.rst

gentlegiantJGC (1):
      Update build_meta.py

liuzhe (1):
      fix version parsing

nullableVoidPtr (2):
      Allow recursive globs for package_data
      Fix mistake on recursive glob test

wim glenn (4):
      make all fields documented as optional in the spec truly optional (dont warn when omitted), and stop coercing missing values to the string "UNKNOWN"
      address review comments
      do not backfill Project-URL: homepage into Home-page: field (causes duplicates on PyPI).  prevent "UNKNOWN" vals from appearing in summary, license, platform.  prevent an extra newline getting added in long description
      fix a typo

Łukasz Daniluk (2):
      Add tests for normalized package name resolution
      Add matching of normalized requirements to canonical packages

谭九鼎 (3):
      Use super()
      Use super()
      Create 3054.misc.rst
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants