ModuleNotFoundError: toml with 'black' hook

[jaraco]

I have this config:

panman master $ cat .pre-commit-config.yaml
repos:
-   repo: https://github.com/ambv/black
    rev: stable
    hooks:
    - id: black
      language_version: python3.6

And I've pip-installed the latest pre-commit 1.12.0 to the Python 3.7.1 environment, but when I run pre-commit, it fails thus:

panman master $ pre-commit run --all-files
black....................................................................Failed
hookid: black

Traceback (most recent call last):
  File "/Users/jaraco/.cache/pre-commit/repoogmwxpko/py_env-python3.6/bin/black", line 7, in <module>
    from black import main
  File "/Users/jaraco/.cache/pre-commit/repoogmwxpko/py_env-python3.6/lib/python3.6/site-packages/black.py", line 41, in <module>
    import toml
ModuleNotFoundError: No module named 'toml'
Traceback (most recent call last):
  File "/Users/jaraco/.cache/pre-commit/repoogmwxpko/py_env-python3.6/bin/black", line 7, in <module>
    from black import main
  File "/Users/jaraco/.cache/pre-commit/repoogmwxpko/py_env-python3.6/lib/python3.6/site-packages/black.py", line 41, in <module>
    import toml
ModuleNotFoundError: No module named 'toml'
Traceback (most recent call last):
  File "/Users/jaraco/.cache/pre-commit/repoogmwxpko/py_env-python3.6/bin/black", line 7, in <module>
    from black import main
  File "/Users/jaraco/.cache/pre-commit/repoogmwxpko/py_env-python3.6/lib/python3.6/site-packages/black.py", line 41, in <module>
    import toml
ModuleNotFoundError: No module named 'toml'
Traceback (most recent call last):
  File "/Users/jaraco/.cache/pre-commit/repoogmwxpko/py_env-python3.6/bin/black", line 7, in <module>
    from black import main
  File "/Users/jaraco/.cache/pre-commit/repoogmwxpko/py_env-python3.6/lib/python3.6/site-packages/black.py", line 41, in <module>
    import toml
ModuleNotFoundError: No module named 'toml'

What am I doing wrong?

[jaraco]

Huh. So it seems that I had to run autoupdate first. After running autoupdate, the same command runs without error. That was unexpected.

panman master $ pre-commit autoupdate
Updating https://github.com/ambv/black...[INFO] Initializing environment for https://github.com/ambv/black.
updating stable -> 18.9b0.
panman master $ pre-commit run --all-files
[INFO] Installing environment for https://github.com/ambv/black.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
black....................................................................Failed
hookid: black

Files were modified by this hook. Additional output:

All done! ✨ 🍰 ✨
136 files left unchanged.
All done! ✨ 🍰 ✨
90 files left unchanged.
reformatted /Users/jaraco/code/yg/G/panman/panman/testing/virtualenv/create
All done! ✨ 🍰 ✨
1 file reformatted, 98 files left unchanged.
All done! ✨ 🍰 ✨
26 files left unchanged.

Does that mean that I had a pre-commit environment lying around from another project that was not compatible with 'black@stable'?

[jaraco]

Oh, interesting. When I ran pre-commit autoupdate, it altered the config file:

panman master $ git diff
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index b9a75affd..61898c735 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -1,6 +1,6 @@
 repos:
 -   repo: https://github.com/ambv/black
-    rev: stable
+    rev: 18.9b0
     hooks:
     - id: black
       language_version: python3.6

Do I want to commit that and push it to the other devs?

[jaraco]

If I revert that change and re-run pre-commit run --all-files, it fails with the toml error again.

[chriskuehl]

I'm not really sure why this would happen with stable, but in general it's recommended to use "immutable" version-based tags instead of a branch (or tag that gets updated, such as stable or master) for pre-commit hooks. The way pre-commit works is that it sets up the environment during the first invocation for a given revision, so by specifying stable you're really getting "whatever the stable version happened to be when you did the install" which is potentially an old version and will differ between environments.

You can see more about the reasoning why it's not recommended to point to a branch at #158 (comment).

When you run autoupdate, it's updating you to what it sees as the latest tag. You could probably look into your pre-commit cache to see which revision it's actually using for stable on your machine. Due to the way this works, it's also possible that wiping out the cache and reinstalling could magically fix it (if stable now points to a different, working revision), though again using a version is probably a better idea in most cases.

[jaraco]

Confirmed - wiping the cache and then running with stable worked around the issue. We'll use concrete versions in the future. Thanks for the considerate and helpful response.

[asottile]

Sorry was away from a computer today :)

It's still odd that there was a version installed that mismatched with the dependencies -- I wonder if black was mistagged at some point with the missing dependency and you happened to install a broken version? 🤔

@chriskuehl's response is spot on too -- there's just a few things I'd add. There's an open issue on black suggesting a change in verbiage around the suggested pre-commit configuration. A few suggestions have been made there but it hasn't really moved forward/backward yet as far as I know. There's also a section in the docs which addresses the mutable refs bits -- of course it's not super helpful until you've known you need to know it (this sentence is a trainwreck but I hope you know what I'm trying to say).

Anyway, roundabout blabbering from me 🙃 -- thanks again for this issue -- very detailed and helpful 🎉