Configurable "build" directory?

[woodruffw]

HI there! Thanks a ton for maintaining this tool; it's a real pleasure to use.

When running python -m build, I can pass --outdir /somewhere/else to control where build will place the completed distribution files (i.e., instead of $SOURCE/dist/).

However build (or something else it uses) still produces a build directory in the source root, with the following structure:

$ ls some-pkg/build
bdist.linux-x86_64
lib

...where lib contains the entire package source tree for some-pkg. This in turn causes problems for a variety of tools in the Python ecosystem:

• pytest automatically discovers both build and the "real" source tree and exits with an error, complaining about duplicate modules.
• mypy attempts to lint both and errors for the same reason (duplicate __init__.pys).
• bellybutton and other QA tools see a "new" source tree, and can't apply their allow/denylists correctly

Each of these has an individual knob/configuration option for fixing the problem, but this causes an N*M configuration issue: a source tree with N Python packages with M tools requires N*M individual configuration changes.

Ideally (for my use case), build would be about to do one of the following:

1. 100% "out-of-tree" builds, by adding an additional option like --builddir to control the placement of the build/ directory.
2. Allow a --cleanup (or similar) option to automatically remove any build directories after all distributions are built

[FFY00]

I think that is a behavior of setuptools, not us.

[woodruffw]

That makes sense; I couldn't find a direct reference to the build directory in build.

It looks like setuptools uses --build-base to pass the build directory; is there a way to plumb that option through here?

[layday]

python -m build -C--global-option=--build-base -C--global-option=<value> might work.

[henryiii]

I'd generally specify the directories for tools - tell pytest to only look in tests, mypy to only look in source (and/or tests), etc. Fighting the normal directory structure is likely to confuse users who are familiar with setuptools but are not familiar with the setup you are trying to enforce. You often don't want mypy looking at your tests, pytest looking at your source, anything looking at your docs, etc.

[tool.mypy]
files = "src"

[tool.pytest.ini_options]
testpaths = [
    "tests",
]

Also, most tools stop at the outer directory, because it doesn't have an __init__.py, so I've never had this be an issue; maybe you are enabling namespace package searches for mypy or something like that?

[woodruffw]

python -m build -C--global-option=--build-base -C--global-option=<value> might work.

No luck, unfortunately; that produces:

usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: option --build-base not recognized

I also tried -C--build-base=/foo/bar, which succeeds but doesn't use the path (silently). I'll keep experimenting, though.

---

Fighting the normal directory structure is likely to confuse users who are familiar with setuptools but are not familiar with the setup you are trying to enforce. You often don't want mypy looking at your tests, pytest looking at your source, anything looking at your docs, etc.

This is mostly a self-inflicted problem on my part, I'll admit 😅. But I work on a large codebase with multiple Python packages in a monorepo, and with each QA/testing/etc. tool enabled and configured on a per-directory basis (i.e., each directly has both some-module/ and test/, at the very least). We do this since a lot of our tools run on both the module and the tests, e.g. we run mypy, bellybutton, and black on everything.

[layday]

Are you sure the setuptools option is called --build-base? Try running it directly with setup.py.

[woodruffw]

Here's what I see with setup.py build --help:

Common commands: (see '--help-commands' for more)

  setup.py build      will build the package underneath 'build/'
  setup.py install    will install the package

Global options:
  --verbose (-v)  run verbosely (default)
  --quiet (-q)    run quietly (turns verbosity off)
  --dry-run (-n)  don't actually do anything
  --help (-h)     show detailed help message
  --no-user-cfg   ignore pydistutils.cfg in your home directory

Options for 'build' command:
  --build-base (-b)  base directory for build library
  --build-purelib    build directory for platform-neutral distributions
  --build-platlib    build directory for platform-specific distributions
  --build-lib        build directory for all distribution (defaults to either
                     build-purelib or build-platlib
  --build-scripts    build directory for scripts
  --build-temp (-t)  temporary build directory
  --plat-name (-p)   platform name to build for, if supported (default: macosx
                     -11.6-x86_64)
  --compiler (-c)    specify the compiler type
  --parallel (-j)    number of parallel build jobs
  --debug (-g)       compile extensions and libraries with debugging
                     information
  --force (-f)       forcibly build everything (ignore file timestamps)
  --executable (-e)  specify final destination interpreter path (build.py)
  --help-compiler    list available compilers

usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

I'll try doing the build directly now and see if one of those other options has an effect.

[woodruffw]

Okay, I confirmed that this:

$ python3 setup.py build --build-base ~/tmp/pybuild

...copies everything into ~/tmp/pybuild. It doesn't produce either a source distribution or a wheel, however.

[woodruffw]

bdist_wheel has the separate --bdist-dir option, but using it like so:

$ python3 setup.py bdist_wheel --bdist-dir ~/tmp/pybuild

...still produces ./build/ (and ./dist/, but that was expected).

[layday]

Well, if the option is only valid for the build command, it won't work with build - the setuptools backend is not able to discriminate between options for different commands.

[rossburton]

I looked into this briefly, and fwiw setuptools' bdist_wheel basically hardcodes ./build.

[layday]

I'll close this since it's not related to build. You can track pypa/wheel#447.