.tmpl files missing in new release .tar.gz

[mgorny]

setuptools-51.1.0.post20201221.tar.gz seems to be missing the .tmpl files. As a result, tests fail:

_____________________________________________________ TestEasyInstallTest.test_script_install _____________________________________________________

self = <setuptools.tests.test_easy_install.TestEasyInstallTest object at 0x000055e0e1c3b088>
sdist_script = '/tmp/portage/dev-python/setuptools-51.1.0_p20201221/temp/pytest-of-portage/pytest-0/test_script_install0/setuptools-test-script-1.0.zip'
tmpdir = local('/tmp/portage/dev-python/setuptools-51.1.0_p20201221/temp/pytest-of-portage/pytest-0/test_script_install0')
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x000055e0e1c3b830>

    @pytest.mark.skipif(not sys.platform.startswith('linux'),
                        reason="Test can only be run on Linux")
    def test_script_install(self, sdist_script, tmpdir, monkeypatch):
        """
        Check scripts are installed.
        """
        dist = Distribution({'script_args': ['easy_install']})
        target = (tmpdir / 'target').ensure_dir()
        cmd = ei.easy_install(
            dist,
            install_dir=str(target),
            args=['x'],
        )
        monkeypatch.setitem(os.environ, 'PYTHONPATH', str(target))
        cmd.ensure_finalized()
>       cmd.easy_install(sdist_script)

setuptools/tests/test_easy_install.py:267: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
setuptools/command/easy_install.py:650: in easy_install
    return self.install_item(None, spec, tmpdir, deps, True)
setuptools/command/easy_install.py:697: in install_item
    self.process_distribution(spec, dist, deps)
setuptools/command/easy_install.py:722: in process_distribution
    self.install_egg_scripts(dist)
setuptools/command/easy_install.py:599: in install_egg_scripts
    dist.get_metadata('scripts/' + script_name)
setuptools/command/easy_install.py:799: in install_script
    body = self._load_template(dev_path) % locals()
setuptools/command/easy_install.py:815: in _load_template
    raw_bytes = resource_string('setuptools', name)
build/test/lib/pkg_resources/__init__.py:1141: in resource_string
    self, resource_name
build/test/lib/pkg_resources/__init__.py:1386: in get_resource_string
    return self._get(self._fn(self.module_path, resource_name))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pkg_resources.DefaultProvider object at 0x00007f679b7580c8>
path = '/tmp/portage/dev-python/setuptools-51.1.0_p20201221/work/setuptools-51.1.0.post20201221-pypy3/setuptools/script.tmpl'

    def _get(self, path):
>       with open(path, 'rb') as stream:
E       FileNotFoundError: [Errno 2] No such file or directory: '/tmp/portage/dev-python/setuptools-51.1.0_p20201221/work/setuptools-51.1.0.post20201221-pypy3/setuptools/script.tmpl'

build/test/lib/pkg_resources/__init__.py:1609: FileNotFoundError
-------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------
Processing setuptools-test-script-1.0.zip
Writing /tmp/portage/dev-python/setuptools-51.1.0_p20201221/temp/easy_install-zph14rrk/setup.cfg
Running setup.py -q bdist_egg --dist-dir /tmp/portage/dev-python/setuptools-51.1.0_p20201221/temp/easy_install-zph14rrk/egg-dist-tmp-z3lf1g7i
Moving setuptools_test_script-1.0-py3.7.egg to /tmp/portage/dev-python/setuptools-51.1.0_p20201221/temp/pytest-of-portage/pytest-0/test_script_install0/target
Adding setuptools-test-script 1.0 to easy-install.pth file
-------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------
warning: install_lib: 'build/lib' does not exist -- no Python modules to install

zip_safe flag not set; analyzing archive contents...

[jaraco]

Oh, that's interesting and unexpected. It's almost certainly related to the move to the skeleton (#2486). It's also a little surprising that the sdist is now a .tar.gz, but not so surprising given that pep517 is likely intentionally disregarding the project's preference.

[jaraco]

Also, FYI, in #2500, that release is releasing with the proper version number (v51.1.0).

[jaraco]

Also, I note this issue might have been caught earlier if not for the blockers in #2318.

[jaraco]

Looking at 957b7a1, I just don't see what change might have caused this new behavior.

[jaraco]

I have confirmed that commit is implicated. Building the package with pep517 on the parent commit (30b66d8), the .tmpl files are included.

[jaraco]

It seems it's this option. Removing it allows the file to be included. Seriously, how is it that including package data would cause files to be excluded? :(

[jaraco]

Looks like this project got bit by #1461.

[webknjaz]

https://setuptools.readthedocs.io/en/latest/userguide/datafiles.html and https://setuptools.readthedocs.io/en/latest/userguide/quickstart.html#including-data-files seem to imply that include_package_data = true only adds files that are present in MANIFEST.in.

[webknjaz]

@jaraco also, I invite you to review https://github.com/pypa/packaging.python.org/pull/761/files.

[jaraco]

I did observe that there's nothing in MANIFEST.in that would include the .tmpl files, but the manifest template did not change... so it seems that the presence of that also cause the exclusion of files that would have been picked up by build_py.data_files. I believe the best quick fix here is to ensure the manifest includes everything, at least until Setuptools can rely on setuptools_scm to resolve files from git metadata.

[webknjaz]

The docs suggest using the package_data dict to include files generated by the build process.

[jaraco]

With this latest commit, I ran pip-run pep517 -- -m pep517.build -s .; cd dist; tar xfz *.gz; rm *.gz; cd *; tox and the tests passed (confirming that the test suite now passes on a build of the sdist). So this issue should be fixed.

[webknjaz]

@jaraco if you suspect this is happening because of the isolation, I suggest trying out python-build's no isolation mode.