Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

prune ** excludes absolutely everything, even if followed by graft and include directives #144

Open
matthewfeickert opened this issue Sep 8, 2021 · 3 comments

Comments

@matthewfeickert
Copy link

matthewfeickert commented Sep 8, 2021

After looking around in the Issues related to prune I think the answer is "no", but I thought I'd ask here explicitly though I think this is just (maybe?) related to Issue #115.

A project I work on has prune ** in our MANIFEST.in, which I'll copy here as well:

$ cat MANIFEST.in 
prune **
graft src

include setup.py
include setup.cfg
include LICENSE
include README.rst
include pyproject.toml
include MANIFEST.in
include AUTHORS

global-exclude __pycache__ *.py[cod]

I've recently noticed that while check-manifest is passing when run here, it isn't doing anything as it finds 0 files

$ check-manifest --version
check-manifest version 0.46
$ check-manifest --verbose
listing source files under version control: 0 files and directories
building an sdist: pyhf-0.6.4.dev8.tar.gz: 0 files and directories
copying source files to a temporary directory
building a clean sdist: pyhf-0.6.4.dev8.tar.gz: 0 files and directories
lists of files in version control and sdist match
$ echo $?
0

(Aside: This might be a nice place to fail out on the user)

However, as (according to the Including files in source distributions with MANIFEST.in PyPA guide)

Setuptools also has undocumented support for ** matching zero or more characters including forward slash, backslash, and colon.

build will happily make an sdist and wheel that has the correct file contents.

$ python -m build --outdir dist/ .
$ python -m tarfile --list dist/*.tar.gz | wc -l
105
$ python -m zipfile --list dist/*.whl | wc -l
81
Full dump from the above:
$ python -m build --outdir dist/ .
$ python -m tarfile --list dist/*.tar.gz
pyhf-0.6.4.dev8/ 
pyhf-0.6.4.dev8/AUTHORS 
pyhf-0.6.4.dev8/LICENSE 
pyhf-0.6.4.dev8/MANIFEST.in 
pyhf-0.6.4.dev8/PKG-INFO 
pyhf-0.6.4.dev8/README.rst 
pyhf-0.6.4.dev8/pyproject.toml 
pyhf-0.6.4.dev8/setup.cfg 
pyhf-0.6.4.dev8/setup.py 
pyhf-0.6.4.dev8/src/ 
pyhf-0.6.4.dev8/src/pyhf/ 
pyhf-0.6.4.dev8/src/pyhf/__init__.py 
pyhf-0.6.4.dev8/src/pyhf/_version.py 
pyhf-0.6.4.dev8/src/pyhf/cli/ 
pyhf-0.6.4.dev8/src/pyhf/cli/__init__.py 
pyhf-0.6.4.dev8/src/pyhf/cli/cli.py 
pyhf-0.6.4.dev8/src/pyhf/cli/complete.py 
pyhf-0.6.4.dev8/src/pyhf/cli/infer.py 
pyhf-0.6.4.dev8/src/pyhf/cli/patchset.py 
pyhf-0.6.4.dev8/src/pyhf/cli/rootio.py 
pyhf-0.6.4.dev8/src/pyhf/cli/spec.py 
pyhf-0.6.4.dev8/src/pyhf/compat.py 
pyhf-0.6.4.dev8/src/pyhf/constraints.py 
pyhf-0.6.4.dev8/src/pyhf/contrib/ 
pyhf-0.6.4.dev8/src/pyhf/contrib/__init__.py 
pyhf-0.6.4.dev8/src/pyhf/contrib/cli.py 
pyhf-0.6.4.dev8/src/pyhf/contrib/utils.py 
pyhf-0.6.4.dev8/src/pyhf/contrib/viz/ 
pyhf-0.6.4.dev8/src/pyhf/contrib/viz/__init__.py 
pyhf-0.6.4.dev8/src/pyhf/contrib/viz/brazil.py 
pyhf-0.6.4.dev8/src/pyhf/data/ 
pyhf-0.6.4.dev8/src/pyhf/data/citation.bib 
pyhf-0.6.4.dev8/src/pyhf/events.py 
pyhf-0.6.4.dev8/src/pyhf/exceptions/ 
pyhf-0.6.4.dev8/src/pyhf/exceptions/__init__.py 
pyhf-0.6.4.dev8/src/pyhf/infer/ 
pyhf-0.6.4.dev8/src/pyhf/infer/__init__.py 
pyhf-0.6.4.dev8/src/pyhf/infer/calculators.py 
pyhf-0.6.4.dev8/src/pyhf/infer/intervals.py 
pyhf-0.6.4.dev8/src/pyhf/infer/mle.py 
pyhf-0.6.4.dev8/src/pyhf/infer/test_statistics.py 
pyhf-0.6.4.dev8/src/pyhf/infer/utils.py 
pyhf-0.6.4.dev8/src/pyhf/interpolators/ 
pyhf-0.6.4.dev8/src/pyhf/interpolators/__init__.py 
pyhf-0.6.4.dev8/src/pyhf/interpolators/code0.py 
pyhf-0.6.4.dev8/src/pyhf/interpolators/code1.py 
pyhf-0.6.4.dev8/src/pyhf/interpolators/code2.py 
pyhf-0.6.4.dev8/src/pyhf/interpolators/code4.py 
pyhf-0.6.4.dev8/src/pyhf/interpolators/code4p.py 
pyhf-0.6.4.dev8/src/pyhf/mixins.py 
pyhf-0.6.4.dev8/src/pyhf/modifiers/ 
pyhf-0.6.4.dev8/src/pyhf/modifiers/__init__.py 
pyhf-0.6.4.dev8/src/pyhf/modifiers/histosys.py 
pyhf-0.6.4.dev8/src/pyhf/modifiers/lumi.py 
pyhf-0.6.4.dev8/src/pyhf/modifiers/normfactor.py 
pyhf-0.6.4.dev8/src/pyhf/modifiers/normsys.py 
pyhf-0.6.4.dev8/src/pyhf/modifiers/shapefactor.py 
pyhf-0.6.4.dev8/src/pyhf/modifiers/shapesys.py 
pyhf-0.6.4.dev8/src/pyhf/modifiers/staterror.py 
pyhf-0.6.4.dev8/src/pyhf/optimize/ 
pyhf-0.6.4.dev8/src/pyhf/optimize/__init__.py 
pyhf-0.6.4.dev8/src/pyhf/optimize/common.py 
pyhf-0.6.4.dev8/src/pyhf/optimize/mixins.py 
pyhf-0.6.4.dev8/src/pyhf/optimize/opt_jax.py 
pyhf-0.6.4.dev8/src/pyhf/optimize/opt_minuit.py 
pyhf-0.6.4.dev8/src/pyhf/optimize/opt_numpy.py 
pyhf-0.6.4.dev8/src/pyhf/optimize/opt_pytorch.py 
pyhf-0.6.4.dev8/src/pyhf/optimize/opt_scipy.py 
pyhf-0.6.4.dev8/src/pyhf/optimize/opt_tflow.py 
pyhf-0.6.4.dev8/src/pyhf/parameters/ 
pyhf-0.6.4.dev8/src/pyhf/parameters/__init__.py 
pyhf-0.6.4.dev8/src/pyhf/parameters/paramsets.py 
pyhf-0.6.4.dev8/src/pyhf/parameters/paramview.py 
pyhf-0.6.4.dev8/src/pyhf/parameters/utils.py 
pyhf-0.6.4.dev8/src/pyhf/patchset.py 
pyhf-0.6.4.dev8/src/pyhf/pdf.py 
pyhf-0.6.4.dev8/src/pyhf/probability.py 
pyhf-0.6.4.dev8/src/pyhf/readxml.py 
pyhf-0.6.4.dev8/src/pyhf/schemas/ 
pyhf-0.6.4.dev8/src/pyhf/schemas/1.0.0/ 
pyhf-0.6.4.dev8/src/pyhf/schemas/1.0.0/defs.json 
pyhf-0.6.4.dev8/src/pyhf/schemas/1.0.0/jsonpatch.json 
pyhf-0.6.4.dev8/src/pyhf/schemas/1.0.0/measurement.json 
pyhf-0.6.4.dev8/src/pyhf/schemas/1.0.0/model.json 
pyhf-0.6.4.dev8/src/pyhf/schemas/1.0.0/patchset.json 
pyhf-0.6.4.dev8/src/pyhf/schemas/1.0.0/workspace.json 
pyhf-0.6.4.dev8/src/pyhf/schemas/HistFactorySchema.dtd 
pyhf-0.6.4.dev8/src/pyhf/simplemodels.py 
pyhf-0.6.4.dev8/src/pyhf/tensor/ 
pyhf-0.6.4.dev8/src/pyhf/tensor/__init__.py 
pyhf-0.6.4.dev8/src/pyhf/tensor/common.py 
pyhf-0.6.4.dev8/src/pyhf/tensor/jax_backend.py 
pyhf-0.6.4.dev8/src/pyhf/tensor/numpy_backend.py 
pyhf-0.6.4.dev8/src/pyhf/tensor/pytorch_backend.py 
pyhf-0.6.4.dev8/src/pyhf/tensor/tensorflow_backend.py 
pyhf-0.6.4.dev8/src/pyhf/utils.py 
pyhf-0.6.4.dev8/src/pyhf/workspace.py 
pyhf-0.6.4.dev8/src/pyhf/writexml.py 
pyhf-0.6.4.dev8/src/pyhf.egg-info/ 
pyhf-0.6.4.dev8/src/pyhf.egg-info/PKG-INFO 
pyhf-0.6.4.dev8/src/pyhf.egg-info/SOURCES.txt 
pyhf-0.6.4.dev8/src/pyhf.egg-info/dependency_links.txt 
pyhf-0.6.4.dev8/src/pyhf.egg-info/entry_points.txt 
pyhf-0.6.4.dev8/src/pyhf.egg-info/requires.txt 
pyhf-0.6.4.dev8/src/pyhf.egg-info/top_level.txt
$ python -m zipfile --list dist/*.whl
File Name                                             Modified             Size
pyhf/__init__.py                               2021-09-01 03:41:30         6148
pyhf/_version.py                               2021-09-08 18:47:12          155
pyhf/compat.py                                 2021-09-01 03:41:30         4036
pyhf/constraints.py                            2021-09-01 03:41:30        10274
pyhf/events.py                                 2021-09-01 03:41:30         4401
pyhf/mixins.py                                 2021-09-01 03:41:30         2048
pyhf/patchset.py                               2021-09-01 03:41:30        11310
pyhf/pdf.py                                    2021-09-01 03:41:30        31029
pyhf/probability.py                            2021-09-01 03:41:30         9537
pyhf/readxml.py                                2021-09-07 03:51:54        13087
pyhf/simplemodels.py                           2021-09-01 03:41:30         5764
pyhf/utils.py                                  2021-09-07 03:51:54         5061
pyhf/workspace.py                              2021-09-07 03:51:54        33903
pyhf/writexml.py                               2021-09-07 03:51:54        10807
pyhf/cli/__init__.py                           2021-09-01 03:41:30          387
pyhf/cli/cli.py                                2021-09-01 03:41:30         1299
pyhf/cli/complete.py                           2021-08-18 14:06:12          982
pyhf/cli/infer.py                              2021-09-01 03:41:30         6605
pyhf/cli/patchset.py                           2021-09-01 03:41:30         4318
pyhf/cli/rootio.py                             2021-09-01 03:41:30         3072
pyhf/cli/spec.py                               2021-09-01 03:41:30        12266
pyhf/contrib/__init__.py                       2021-08-18 14:06:12           89
pyhf/contrib/cli.py                            2021-09-01 03:41:30         2051
pyhf/contrib/utils.py                          2021-09-01 03:41:30         2674
pyhf/contrib/viz/__init__.py                   2021-08-18 14:06:12           49
pyhf/contrib/viz/brazil.py                     2021-09-01 03:41:30        13476
pyhf/data/citation.bib                         2021-09-07 03:51:54          699
pyhf/exceptions/__init__.py                    2021-09-01 03:41:30         4784
pyhf/infer/__init__.py                         2021-09-01 03:41:30         8239
pyhf/infer/calculators.py                      2021-09-01 03:41:30        35522
pyhf/infer/intervals.py                        2021-09-01 03:41:30         2647
pyhf/infer/mle.py                              2021-09-01 03:41:30         8281
pyhf/infer/test_statistics.py                  2021-09-01 03:41:30        15867
pyhf/infer/utils.py                            2021-09-01 03:41:30         3336
pyhf/interpolators/__init__.py                 2021-09-01 03:41:30         1644
pyhf/interpolators/code0.py                    2021-09-01 03:41:30         3847
pyhf/interpolators/code1.py                    2021-09-01 03:41:30         4498
pyhf/interpolators/code2.py                    2021-09-01 03:41:30         5349
pyhf/interpolators/code4.py                    2021-09-01 03:41:30        16006
pyhf/interpolators/code4p.py                   2021-09-01 03:41:30         4954
pyhf/modifiers/__init__.py                     2021-09-01 03:41:30         7242
pyhf/modifiers/histosys.py                     2021-09-01 03:41:30         3290
pyhf/modifiers/lumi.py                         2021-09-01 03:41:30         2539
pyhf/modifiers/normfactor.py                   2021-09-01 03:41:30         2727
pyhf/modifiers/normsys.py                      2021-09-01 03:41:30         3312
pyhf/modifiers/shapefactor.py                  2021-09-01 03:41:30         5889
pyhf/modifiers/shapesys.py                     2021-09-07 14:57:40         7041
pyhf/modifiers/staterror.py                    2021-09-01 03:41:30         5274
pyhf/optimize/__init__.py                      2021-09-01 03:41:30         1493
pyhf/optimize/common.py                        2021-09-01 03:41:30         5542
pyhf/optimize/mixins.py                        2021-09-01 03:41:30         7237
pyhf/optimize/opt_jax.py                       2021-09-01 03:41:30         2491
pyhf/optimize/opt_minuit.py                    2021-09-01 03:41:30         5357
pyhf/optimize/opt_numpy.py                     2021-09-01 03:41:30         1051
pyhf/optimize/opt_pytorch.py                   2021-09-01 03:41:30         1353
pyhf/optimize/opt_scipy.py                     2021-09-01 03:41:30         3434
pyhf/optimize/opt_tflow.py                     2021-09-01 03:41:30         1583
pyhf/parameters/__init__.py                    2021-09-01 03:41:30          447
pyhf/parameters/paramsets.py                   2021-09-01 03:41:30         2018
pyhf/parameters/paramview.py                   2021-09-01 03:41:30         2877
pyhf/parameters/utils.py                       2021-09-01 03:41:30         2820
pyhf/schemas/HistFactorySchema.dtd             2021-01-21 19:23:20         6521
pyhf/schemas/1.0.0/defs.json                   2021-09-01 03:41:30        13069
pyhf/schemas/1.0.0/jsonpatch.json              2021-08-18 14:06:12          180
pyhf/schemas/1.0.0/measurement.json            2021-08-18 14:06:12          184
pyhf/schemas/1.0.0/model.json                  2021-08-18 14:06:12          172
pyhf/schemas/1.0.0/patchset.json               2021-08-18 14:06:12          178
pyhf/schemas/1.0.0/workspace.json              2021-08-18 14:06:12          180
pyhf/tensor/__init__.py                        2021-09-01 03:41:30         2041
pyhf/tensor/common.py                          2021-09-01 03:41:30         3278
pyhf/tensor/jax_backend.py                     2021-09-07 03:51:54        18044
pyhf/tensor/numpy_backend.py                   2021-09-02 17:02:22        17054
pyhf/tensor/pytorch_backend.py                 2021-09-02 17:02:22        18220
pyhf/tensor/tensorflow_backend.py              2021-09-07 03:51:54        23064
pyhf-0.6.4.dev8.dist-info/LICENSE              2021-09-08 18:47:12        10760
pyhf-0.6.4.dev8.dist-info/METADATA             2021-09-08 18:47:12        21377
pyhf-0.6.4.dev8.dist-info/WHEEL                2021-09-08 18:47:12           92
pyhf-0.6.4.dev8.dist-info/entry_points.txt     2021-09-08 18:47:12           39
pyhf-0.6.4.dev8.dist-info/top_level.txt        2021-09-08 18:47:12            5
pyhf-0.6.4.dev8.dist-info/RECORD               2021-09-08 18:47:12         6457

Given Issue #115, does this mean check-manifest is incompatible with this syntax and to use check-manifest users should manually add / exclude all top level files?

cc @alexander-held

@mgedmin
Copy link
Owner

mgedmin commented Sep 9, 2021

This is a bug, or, more precisely, a missing feature: check-manifest currently processes exclude directives (such as prune) and ignores include directives.

I would like to fix it some day.

@mgedmin mgedmin added the bug label Sep 9, 2021
@mgedmin mgedmin changed the title Does check-manifest support ** like setuptools does? prune ** excludes absolutely everything, even if followed by graft and include directives Sep 9, 2021
@matthewfeickert
Copy link
Author

This is a bug, or, more precisely, a missing feature: check-manifest currently processes exclude directives (such as prune) and ignores include directives.

Thanks for the very responsive reply @mgedmin — truly appreciate it! This information is helpful, so thanks for clarifying the situation.

I would like to fix it some day.

I assume that this is nontrivial and not something that is an appropriate contribution for a first time contributor to the project. But, if I was selfishly interested in trying to help address this, do you have any advice on where to start working? I should of course also just start looking myself.

@mgedmin
Copy link
Owner

mgedmin commented Sep 10, 2021

I assume that this is nontrivial and not something that is an appropriate contribution for a first time contributor to the project. But, if I was selfishly interested in trying to help address this, do you have any advice on where to start working? I should of course also just start looking myself.

I have this IgnoreList class that currently collects all the exclusion rules. It needs to be extended to handle include rules in proper order and renamed to something like RuleList.

Then read_manifest()'s inner helpers need to be changed (and renamed) so they handle all the include directives (there's an XXX comment) and return a RuleList instead of an IgnoreList.

Then I have to figure out how ignore rules specified in check-manifest's configuration interact with MANIFEST's include/exclude rules, order-wise, and maybe update the += statement in check_manifest().

And, of course, unit tests for all of the above. Oh dear, I haven't migrated to pytest yet, that's a shame. Well, the end-of-life of Python 3.5 makes things easier there -- the Windows console issue that was a blocker should no longer be a blocker. Eh, maybe unittest isn't that bad.

The highest risk is that I might be too burned out to review a PR in a timely fashion. (I'm currently failing to review three other PR's that are sitting there in open browser tabs since July.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants