Allow envars and shell commands usage in "include" path

[jaklan]

Let's imagine a monorepo structure like:

pyproject.toml
projects/
-- foobar/
---- pyproject.toml

In the root pyproject.toml we define monorepo-wide commands, which are imported in all project-level pyproject.tomls.

To import the global commands, in projects/foobar/pyproject.toml we need to specify:

[tool.poe]
include = "../../pyproject.toml" 

which is pretty inconvenient and error-prone:

• it's very easy to specify a wrong file when you refer to a parent directory a few levels up

• a repo / project structure can change and you have to remember to update the path

  • it's tricky to notice a bug in such situation because of:

    If a referenced file is missing then poe ignores it without error, though failure to read the contents will result in failure.

  • by coincidence you can even import a wrong file

• reusing / templating the config across different projects is harder because they can be nested at different levels

I wish it could be possible to add a support for envars and shell commands in include, so we could rely on a more robust solution like:

[tool.poe]
include = "$(git rev-parse --show-toplevel)/pyproject.toml" 

[nat-n]

Hi @jaklan, thanks for the feedback. I see how this would be useful.

I'm currently working on a bit of a refactor of config management works, which adds support for referencing env vars in the include path. So maybe exposing something like ${POE_GIT_ROOT} would do the trick there?

Supporting sub-shell syntax would be a bit of a different story.

[jaklan]

@nat-n thanjs for the quick response! Afaik git doesn't expose any envar to identify the repo root, but if you provide it as a custom poe var - I believe it should be good enough for many use-cases (incl. ours).

We would just like to avoid a situation when it's needed to rely on an envar which has to be set locally by all developers.

[nat-n]

Hi @jaklan, I was thinking about exactly how to implement this, as a lazily resolve environment variable set by poe, and I realised the problem isn't as it seemed.

Firstly, how exactly should POE_GIT_ROOT be defined by poe? If it's relative to the present working directory then that won't always be inside the project making it brittle. If it's relative to the relevant config file then it gets complicated if you're composing your project with git modules. And if it's relative to the root config file, then you probably already have the answer you care about with the POE_ROOT environment variable without having to consult with git.

What do you think the logic should be to be robust?

EDIT: maybe the way to be consistent despite submodules is git rev-parse --show-superproject-working-tree --show-toplevel ?

[jaklan]

@nat-n

I just thought about triggering $(git rev-parse --show-toplevel) in current (or specified) working directory, as an alternative to specifying a shell command directly in include. I won't really help with the submodule case, I don't use them intentionally.

About POE_ROOT:

POE_ROOT: path to the parent directory of the main tasks file (e.g. pyproject.toml).

If we have the structure like:

pyproject.toml
projects/
-- pyproject.toml
-- foobar/
---- pyproject.toml

and define:

[tool.poe]
include = "$POE_ROOT/pyproject.toml" 

in projects/foobar/pyproject.toml - would it refer to the root directory, to projects/, or to projects/foobar? How to understand "the main tasks file" here?