Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

changedir breaks script discovery #2089

Closed
felix-hilden opened this issue Jun 17, 2021 · 15 comments
Closed

changedir breaks script discovery #2089

felix-hilden opened this issue Jun 17, 2021 · 15 comments
Labels
type:question ❔ a question about how things work or if something is a bug or a feature

Comments

@felix-hilden
Copy link

Hi, firstly thanks for providing Tox. It's awesome! I also saw the PyCon talk about Tox 4 and I'm really looking forward to it 😄

I've created an env for building Sphinx documentation with its make scripts. I know the documentation suggests using sphinx-build directly but anyways: the script cannot be found under the directory.

InvocationError for command could not find executable my_script

Output of tox -e script -rvv

(venv) C:\Felix\Git\ztst>tox -e script -rvv
using tox.ini: C:\Felix\Git\ztst\tox.ini (pid 14348)
  removing C:\Felix\Git\ztst\.tox\log
using tox-3.23.1 from c:\felix\git\ztst\venv\lib\site-packages\tox\__init__.py (pid 14348)
GLOB start: packaging 
GLOB sdist-make: C:\Felix\Git\ztst\setup.py
  removing C:\Felix\Git\ztst\.tox\dist
[6856] C:\Felix\Git\ztst$ 'c:\felix\git\ztst\venv\scripts\python.exe' setup.py sdist --formats=zip --dist-dir '.tox\dist' >.tox\log
\GLOB-0.log
running sdist
running egg_info
writing package.egg-info\PKG-INFO
writing dependency_links to package.egg-info\dependency_links.txt
writing top-level names to package.egg-info\top_level.txt
reading manifest file 'package.egg-info\SOURCES.txt'
writing manifest file 'package.egg-info\SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md

running check
warning: check: missing required meta-data: url

creating package-0.0.1
creating package-0.0.1\package
creating package-0.0.1\package.egg-info
copying files to package-0.0.1...
copying setup.py -> package-0.0.1
copying package\__init__.py -> package-0.0.1\package
copying package.egg-info\PKG-INFO -> package-0.0.1\package.egg-info
copying package.egg-info\SOURCES.txt -> package-0.0.1\package.egg-info
copying package.egg-info\dependency_links.txt -> package-0.0.1\package.egg-info
copying package.egg-info\top_level.txt -> package-0.0.1\package.egg-info
Writing package-0.0.1\setup.cfg
creating '.tox\dist\package-0.0.1.zip' and adding 'package-0.0.1' to it
adding 'package-0.0.1'
adding 'package-0.0.1\package'
adding 'package-0.0.1\package.egg-info'
adding 'package-0.0.1\PKG-INFO'
adding 'package-0.0.1\setup.cfg'
adding 'package-0.0.1\setup.py'
adding 'package-0.0.1\package\__init__.py'
adding 'package-0.0.1\package.egg-info\dependency_links.txt'
adding 'package-0.0.1\package.egg-info\PKG-INFO'
adding 'package-0.0.1\package.egg-info\SOURCES.txt'
adding 'package-0.0.1\package.egg-info\top_level.txt'
removing 'package-0.0.1' (and everything under it)

GLOB finish: packaging  after 0.28 seconds
copying new sdistfile to 'C:\\Users\\Felix The III\\.tox\\distshare\\package-0.0.1.zip'
package .tmp\package\1\package-0.0.1.zip links to dist\package-0.0.1.zip (C:\Felix\Git\ztst\.tox)
c:\felix\git\ztst\venv\scripts\python.exe (c:\felix\git\ztst\venv\scripts\python.exe) is {'executable': 'c:\\felix\\git\\ztst\\venv
\\scripts\\python.exe', 'implementation': 'CPython', 'version_info': [3, 9, 4, 'final', 0], 'version': '3.9.4 (tags/v3.9.4:1f2e308,
 Apr  6 2021, 13:40:21) [MSC v.1928 64 bit (AMD64)]', 'is_64': True, 'sysplatform': 'win32', 'os_sep': '\\', 'extra_version_info':
None}
script uses c:\felix\git\ztst\venv\scripts\python.exe
script start: getenv C:\Felix\Git\ztst\.tox\script
script cannot reuse: -r flag
script recreate: C:\Felix\Git\ztst\.tox\script
  removing C:\Felix\Git\ztst\.tox\script
setting PATH=C:\Felix\Git\ztst\.tox\script\Scripts;C:\Felix\Git\ztst\venv\Scripts;C:\Program Files\Python39\Scripts\;C:\Program Fil
es\Python39\;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin;C:\Program Files\R\Rtools\bin;C:\Program Files\Python37\S
cripts\;C:\Program Files\Python37\;C:\Program Files (x86)\Common Files\Intel\Shared Libraries\redist\intel64\compiler;C:\Program Fi
les (x86)\Common Files\Intel\Shared Libraries\redist\ia32\compiler;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windo
ws\System32\WindowsPowerShell\v1.0\;C:\Users\Felix The III\.dnx\bin;C:\Program Files\Microsoft DNX\Dnvm\;C:\Program Files\Microsoft
 SQL Server\130\Tools\Binn\;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit\;C:\Program Files (x86)\NVIDIA Corp
oration\PhysX\Common;C:\Program Files\MiKTeX 2.9\miktex\bin\x64\;C:\Program Files (x86)\WinSCP\;C:\WINDOWS\system32;C:\WINDOWS;C:\W
INDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\Meld;C:\Progra
m Files\Java\jdk1.8.0_131\jre\bin;C:\Program Files\NVIDIA Corporation\Nsight Compute 2019.5.0\;C:\Ruby26-x64\Perl\c\bin;C:\Ruby26-x
64\Perl\perl\site\bin;C:\Ruby26-x64\Perl\perl\bin;C:\Program Files\MATLAB\R2020a\bin;C:\Program Files\PuTTY\;C:\Program Files\Inksc
ape\bin;C:\Program Files\dotnet\;C:\Program Files (x86)\Graphviz\bin;C:\Program Files\Git\bin;;C:\WINDOWS\system32;C:\WINDOWS;C:\WI
NDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Ruby26-x64\bin;C:\Users\Felix The I
II\AppData\Local\Microsoft\WindowsApps;C:\Users\Felix The III\AppData\Local\Microsoft\WindowsApps
[12584] C:\Felix\Git\ztst\.tox$ 'c:\felix\git\ztst\venv\scripts\python.exe' -m virtualenv --no-download --python 'c:\felix\git\ztst
\venv\scripts\python.exe' script
created virtual environment CPython3.9.4.final.0-64 in 443ms
  creator CPython3Windows(dest=C:\Felix\Git\ztst\.tox\script, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\Felix The III\App
Data\Local\pypa\virtualenv)
    added seed packages: pip==21.1.2, setuptools==57.0.0, wheel==0.36.2
  activators BashActivator,BatchActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
script finish: getenv C:\Felix\Git\ztst\.tox\script after 0.89 seconds
script start: installpkg C:\Felix\Git\ztst\.tox\.tmp\package\1\package-0.0.1.zip
script inst: C:\Felix\Git\ztst\.tox\.tmp\package\1\package-0.0.1.zip
write config to C:\Felix\Git\ztst\.tox\script\.tox-config1 as 'a1b7ef8ecc300c0c3968e75eee0bacebceafee09837fc27d41da46c09ea194b3 c:\
\felix\\git\\ztst\\venv\\scripts\\python.exe\n3.23.1 0 0 0'
setting PATH=C:\Felix\Git\ztst\.tox\script\Scripts;C:\Felix\Git\ztst\venv\Scripts;C:\Program Files\Python39\Scripts\;C:\Program Fil
es\Python39\;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin;C:\Program Files\R\Rtools\bin;C:\Program Files\Python37\S
cripts\;C:\Program Files\Python37\;C:\Program Files (x86)\Common Files\Intel\Shared Libraries\redist\intel64\compiler;C:\Program Fi
les (x86)\Common Files\Intel\Shared Libraries\redist\ia32\compiler;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windo
ws\System32\WindowsPowerShell\v1.0\;C:\Users\Felix The III\.dnx\bin;C:\Program Files\Microsoft DNX\Dnvm\;C:\Program Files\Microsoft
 SQL Server\130\Tools\Binn\;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit\;C:\Program Files (x86)\NVIDIA Corp
oration\PhysX\Common;C:\Program Files\MiKTeX 2.9\miktex\bin\x64\;C:\Program Files (x86)\WinSCP\;C:\WINDOWS\system32;C:\WINDOWS;C:\W
INDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\Meld;C:\Progra
m Files\Java\jdk1.8.0_131\jre\bin;C:\Program Files\NVIDIA Corporation\Nsight Compute 2019.5.0\;C:\Ruby26-x64\Perl\c\bin;C:\Ruby26-x
64\Perl\perl\site\bin;C:\Ruby26-x64\Perl\perl\bin;C:\Program Files\MATLAB\R2020a\bin;C:\Program Files\PuTTY\;C:\Program Files\Inksc
ape\bin;C:\Program Files\dotnet\;C:\Program Files (x86)\Graphviz\bin;C:\Program Files\Git\bin;;C:\WINDOWS\system32;C:\WINDOWS;C:\WI
NDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Ruby26-x64\bin;C:\Users\Felix The I
II\AppData\Local\Microsoft\WindowsApps;C:\Users\Felix The III\AppData\Local\Microsoft\WindowsApps
[6656] C:\Felix\Git\ztst$ 'C:\Felix\Git\ztst\.tox\script\Scripts\python.EXE' -m pip install --exists-action w '.tox\.tmp\package\1\
package-0.0.1.zip'
Processing c:\felix\git\ztst\.tox\.tmp\package\1\package-0.0.1.zip
Building wheels for collected packages: package
  Building wheel for package (setup.py) ... done
  Created wheel for package: filename=package-0.0.1-py3-none-any.whl size=1299 sha256=2b23f54312efcb9ccde8fdc40a2e0bde06b6699d9c261
ddec7add086f590c4b9
  Stored in directory: c:\users\felix the iii\appdata\local\pip\cache\wheels\3a\ff\d7\7610bc53c7eb3c2f1bbde735f922ecaad85b79d792533
8629c
Successfully built package
Installing collected packages: package
Successfully installed package-0.0.1
script finish: installpkg C:\Felix\Git\ztst\.tox\.tmp\package\1\package-0.0.1.zip after 2.81 seconds
script start: envreport 
setting PATH=C:\Felix\Git\ztst\.tox\script\Scripts;C:\Felix\Git\ztst\venv\Scripts;C:\Program Files\Python39\Scripts\;C:\Program Fil
es\Python39\;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin;C:\Program Files\R\Rtools\bin;C:\Program Files\Python37\S
cripts\;C:\Program Files\Python37\;C:\Program Files (x86)\Common Files\Intel\Shared Libraries\redist\intel64\compiler;C:\Program Fi
les (x86)\Common Files\Intel\Shared Libraries\redist\ia32\compiler;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windo
ws\System32\WindowsPowerShell\v1.0\;C:\Users\Felix The III\.dnx\bin;C:\Program Files\Microsoft DNX\Dnvm\;C:\Program Files\Microsoft
 SQL Server\130\Tools\Binn\;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit\;C:\Program Files (x86)\NVIDIA Corp
oration\PhysX\Common;C:\Program Files\MiKTeX 2.9\miktex\bin\x64\;C:\Program Files (x86)\WinSCP\;C:\WINDOWS\system32;C:\WINDOWS;C:\W
INDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\Meld;C:\Progra
m Files\Java\jdk1.8.0_131\jre\bin;C:\Program Files\NVIDIA Corporation\Nsight Compute 2019.5.0\;C:\Ruby26-x64\Perl\c\bin;C:\Ruby26-x
64\Perl\perl\site\bin;C:\Ruby26-x64\Perl\perl\bin;C:\Program Files\MATLAB\R2020a\bin;C:\Program Files\PuTTY\;C:\Program Files\Inksc
ape\bin;C:\Program Files\dotnet\;C:\Program Files (x86)\Graphviz\bin;C:\Program Files\Git\bin;;C:\WINDOWS\system32;C:\WINDOWS;C:\WI
NDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Ruby26-x64\bin;C:\Users\Felix The I
II\AppData\Local\Microsoft\WindowsApps;C:\Users\Felix The III\AppData\Local\Microsoft\WindowsApps
[12348] C:\Felix\Git\ztst$ 'C:\Felix\Git\ztst\.tox\script\Scripts\python.EXE' -m pip freeze >.tox\script\log\script-0.log
script finish: envreport  after 0.40 seconds
script installed: package @ file:///C:/Felix/Git/ztst/.tox/.tmp/package/1/package-0.0.1.zip
script start: run-test-pre 
script run-test-pre: PYTHONHASHSEED='551'
script finish: run-test-pre  after 0.00 seconds
script start: run-test 
script run-test: commands[0] | my_script
setting PATH=C:\Felix\Git\ztst\.tox\script\Scripts;C:\Felix\Git\ztst\venv\Scripts;C:\Program Files\Python39\Scripts\;C:\Program Fil
es\Python39\;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin;C:\Program Files\R\Rtools\bin;C:\Program Files\Python37\S
cripts\;C:\Program Files\Python37\;C:\Program Files (x86)\Common Files\Intel\Shared Libraries\redist\intel64\compiler;C:\Program Fi
les (x86)\Common Files\Intel\Shared Libraries\redist\ia32\compiler;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windo
ws\System32\WindowsPowerShell\v1.0\;C:\Users\Felix The III\.dnx\bin;C:\Program Files\Microsoft DNX\Dnvm\;C:\Program Files\Microsoft
 SQL Server\130\Tools\Binn\;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit\;C:\Program Files (x86)\NVIDIA Corp
oration\PhysX\Common;C:\Program Files\MiKTeX 2.9\miktex\bin\x64\;C:\Program Files (x86)\WinSCP\;C:\WINDOWS\system32;C:\WINDOWS;C:\W
INDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\Meld;C:\Progra
m Files\Java\jdk1.8.0_131\jre\bin;C:\Program Files\NVIDIA Corporation\Nsight Compute 2019.5.0\;C:\Ruby26-x64\Perl\c\bin;C:\Ruby26-x
64\Perl\perl\site\bin;C:\Ruby26-x64\Perl\perl\bin;C:\Program Files\MATLAB\R2020a\bin;C:\Program Files\PuTTY\;C:\Program Files\Inksc
ape\bin;C:\Program Files\dotnet\;C:\Program Files (x86)\Graphviz\bin;C:\Program Files\Git\bin;;C:\WINDOWS\system32;C:\WINDOWS;C:\WI
NDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Ruby26-x64\bin;C:\Users\Felix The I
II\AppData\Local\Microsoft\WindowsApps;C:\Users\Felix The III\AppData\Local\Microsoft\WindowsApps
ERROR: InvocationError for command could not find executable my_script
script finish: run-test  after 0.04 seconds
script start: run-test-post 
script finish: run-test-post  after 0.00 seconds
____________________________________________________________ summary _____________________________________________________________
ERROR:   script: commands failed
cleanup C:\Felix\Git\ztst\.tox\.tmp\package\1\package-0.0.1.zip

Environment: Windows 10, Python 3.9.4, virtualenv, tox 3.23.1

; my_script.bat
echo YEAH.
; tox.ini
[testenv:script]
changedir = doc
allowlist_externals = my_script
commands = my_script

I also had a dummy project with a setup.py file just for installing the package.

  • When the script is in the top level folder (and there's no changedir), this works
  • This works on Linux (conf, workflow, successful run)
  • I'm pretty sure this has worked on Windows before

For the reasons above, I posted this as a bug. But feel free to correct me if I'm simply doing something wrong!

@felix-hilden felix-hilden added the bug:normal affects many people or has quite an impact label Jun 17, 2021
@gaborbernat gaborbernat added type:question ❔ a question about how things work or if something is a bug or a feature and removed bug:normal affects many people or has quite an impact labels Jun 17, 2021
@gaborbernat
Copy link
Member

I know the documentation suggests using sphinx-build directly

Yeah, you definitely want to do this.

the script cannot be found under the directory.

The script is my_script.bat, not my_script. Note Windows command lines auto-fallback to BAT, PS1 and so on extensions, however, tox being platform-agnostic offers no such service.

@gaborbernat
Copy link
Member

  • I'm pretty sure this has worked on Windows before

Can you post a tox 3 run that shows it works?

@felix-hilden
Copy link
Author

Thanks for the prompt response! This also happens with

[testenv:script]
changedir = doc
allowlist_externals = my_script.bat
commands = my_script.bat

And I want to reiterate that it worked without the extension when changedir had no value. Sadly I could not produce the working configuration anymore when trying various combinations of allow and command, but I can't imagine why I would have built the script and managed to run it for at least a year without noticing 😅 But certainly it's a possibility.

@felix-hilden
Copy link
Author

Another thing came to mind: it of course works on Linux because make is not the batch script, it's an actual program. My bad on that one.

@gaborbernat gaborbernat added bug:normal affects many people or has quite an impact and removed type:question ❔ a question about how things work or if something is a bug or a feature labels Jun 17, 2021
@gaborbernat gaborbernat added this to the 4.0 milestone Jun 17, 2021
@gaborbernat gaborbernat changed the title Cannot find batch script with changedir on Windows changedir breaks script discovery Jun 17, 2021
@gaborbernat
Copy link
Member

So just to confirm the script above works on tox 3 but doesn't with tox 4?

@felix-hilden
Copy link
Author

I started narrowing it down a bit, but couldn't find a version that works.

  • tox==3.14.6, which is quite old: doesn't work
  • tox==3.23.1, which is the latest, as in the original post: doesn't work

I'll try the 4.0 branch next.

@gaborbernat gaborbernat removed this from the 4.0 milestone Jun 17, 2021
@gaborbernat gaborbernat added type:question ❔ a question about how things work or if something is a bug or a feature and removed bug:normal affects many people or has quite an impact labels Jun 17, 2021
@gaborbernat
Copy link
Member

Ah, ok then. The scripts are executed and discovered after the changedir is executed. If you do changedir you'd want to specify scripts relative, or better don't really on relative scripts, make them absolute:

[testenv:script]
changedir = doc
commands = {toxinidir}{/}my_script.bat

@felix-hilden
Copy link
Author

felix-hilden commented Jun 17, 2021

Hmm okay, I can get it to work sometimes, but not in the case you presented:

; Works:
changedir = doc
allowlist_externals = my_script
commands = doc/my_script

; Works:
commands = doc/my_script

; Does NOT work:
changedir = doc
commands = {toxinidir}{/}my_script.bat
; results in InvocationError, exit code 2 and the path references the top dir, not doc

; Works:
changedir = doc
commands = {toxinidir}{/}doc/my_script.bat

; Works:
commands = {toxinidir}{/}doc/my_script.bat

;  Does NOT work:
changedir = doc
commands = {toxinidir}{/}doc/my_script
; InvocationError, exit code 2 and the path is weird: ...\top\doc/my_script

Should this information be included in the documentation then? I was very confused, and this helped a lot. Although I don't fully understand the difference between the various runs above, I think at least including something about running scripts would be helpful. I guess the ones without changedir are just a matter of working directory. In any case, thank you very much for the help! 🎉

@felix-hilden
Copy link
Author

felix-hilden commented Jun 17, 2021

Actually, reading your last comment, what do you mean by relative exactly? Because now they need to be relative to the initial directory, even though they are discovered after executing the changedir. I'd expect changedir to change the directory which the scripts are relative to.

@gaborbernat
Copy link
Member

Seems scripts are discovered relative to toxinidir, not the cwd (which is altered by changedir). We can't really change this without breaking a lot of people. In general though my recommendation is to use the {toxinidir}{/}doc{/}my_script.bat format, as that's the most robust. We could document some of these.

@felix-hilden
Copy link
Author

I understand that, and it's an acceptable solution to me. But would it be a possibility in Tox 4?

I'd be happy to contribute such documentation if you'd accept it!

@gaborbernat
Copy link
Member

For sure, though tox 4 documentation is still WIP and I'm currently caught up with doing a POC for PEP-662, so might take a few weeks until we're ready to accept that.

@felix-hilden
Copy link
Author

I meant documentation for Tox 3 for now. Then depending on whether this is changed or left as it is in Tox 4 is up to you, and the documentation can be moved over or updated. How does that sound?

felix-hilden added a commit to felix-hilden/tox that referenced this issue Jun 17, 2021
felix-hilden added a commit to felix-hilden/tox that referenced this issue Jun 18, 2021
gaborbernat pushed a commit to gaborbernat/tox that referenced this issue Jun 22, 2021
gaborbernat added a commit that referenced this issue Jun 23, 2021
Co-authored-by: Felix Hildén <felix.hilden@gmail.com>
Co-authored-by: felix-hilden <felix.hilden@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
@felix-hilden
Copy link
Author

I think this can be closed now!

@gaborbernat
Copy link
Member

Thanks for your contribution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type:question ❔ a question about how things work or if something is a bug or a feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants