Jupyter notebook support (#2357)

To summarise, based on what was discussed in that issue:

due to not being able to parse automagics (e.g. pip install black)
without a running IPython kernel, cells with syntax which is parseable
by neither ast.parse nor IPython will be skipped cells with multiline
magics will be skipped trailing semicolons will be preserved, as they
are often put there intentionally in Jupyter Notebooks to suppress
unnecessary output

Commit history before merge (excluding merge commits):

* wip
* fixup tests
* skip tests if no IPython
* install test requirements in ipynb tests
* if --ipynb format all as ipynb
* wip
* add some whole-notebook tests
* docstrings
* skip multiline magics
* add test for nested cell magic
* remove ipynb_test.yml, put ipynb tests in tox.ini
* add changelog entry
* typo
* make token same length as magic it replaces
* only include .ipynb by default if jupyter dependencies are found
* remove logic from const
* fixup
* fixup
* re.compile
* noop
* clear up
* new_src -> dst
* early exit for non-python notebooks
* add non-python test notebook
* add repo with many notebooks to black-primer
* install extra dependencies for black-primer
* fix planetary computer examples url
* dont run on ipynb files by default
* add scikit-lego (Expected to change) to black-primer
* add ipynb-specific diff
* fixup
* run on all (including ipynb) by default
* remove --include .ipynb from scikit-lego black-primer
* use tokenize so as to mirror the exact logic in IPython.core.displayhooks quiet
* fixup
* 🎨
* clarify docstring
* add test for when comment is after trailing semicolon
* enumerate(reversed) instead of [::-1]
* clarify docstrings
* wip
* use jupyter and no_jupyter marks
* use THIS_DIR
* windows fixup
* perform safe check cell-by-cell for ipynb
* only perform safe check in ipynb if not fast
* remove redundant Optional
* 🎨
* use typeguard
* dont process cell containing transformed magic
* require typing extensions before 3.10 so as to have TypeGuard
* use dataclasses
* mention black[jupyter] in docs as well as in README
* add faq
* add message to assertion error
* add test for indented quieted cell
* use tokenize_rt else we cant roundtrip
* fmake fronzet set for tokens to ignore when looking for trailing semicolon
* remove planetary code examples as recent commits result in changes
* use dataclasses which inherit from ast.NodeVisitor
* bump typing-extensions so that TypeGuard is available
* bump typing-extensions in Pipfile
* add test with notebook with empty metadata
* pipenv lock
* deprivative validate_cell
* Update README.md
* Update docs/getting_started.md
* dont cache notebooks if jupyter dependencies arent found
* dont write to cache if jupyter deps are not installed
* add notebook which cant be parsed
* use clirunner
* remove other subprocess calls
* add docstring
* make verbose and quiet keyword only
* 🎨
* run second many test on directory, not on file
* test for warning message when running on directory
* early return from non-python cell magics
* move NothingChanged to report to avoid circular import
* remove circular import
* reinstate --ipynb flag

Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>

.github/workflows/primer.yml

@@ -38,7 +38,7 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          python -m pip install -e ".[d]"
+          python -m pip install -e ".[d,jupyter]"
 
       - name: Primer run
         env:

.gitignore

@@ -18,3 +18,4 @@ src/_black_version.py
 *.swp
 .hypothesis/
 venv/
+.ipynb_checkpoints/

.pre-commit-hooks.yaml

@@ -7,3 +7,14 @@
   minimum_pre_commit_version: 2.9.2
   require_serial: true
   types_or: [python, pyi]
+- id: black-jupyter
+  name: black-jupyter
+  description:
+    "Black: The uncompromising Python code formatter (with Jupyter Notebook support)"
+  entry: black
+  language: python
+  language_version: python3
+  minimum_pre_commit_version: 2.9.2
+  require_serial: true
+  types_or: [python, pyi, jupyter]
+  additional_dependencies: [".[jupyter]"]

CHANGES.md

@@ -2,7 +2,10 @@
 
 ## Unreleased
 
-- Moved from `appdirs` dependency to `platformdirs` (#2375)
+### _Black_
+
+- Add support for formatting Jupyter Notebook files (#2357)
+- Move from `appdirs` dependency to `platformdirs` (#2375)
 
 ## 21.7b0
 

Pipfile

@@ -33,6 +33,6 @@ pathspec = ">=0.8.1"
 regex = ">=2020.1.8"
 tomli = ">=0.2.6, <2.0.0"
 typed-ast = "==1.4.2"
-typing_extensions = {"python_version <" = "3.8","version >=" = "3.7.4"}
+typing_extensions = {"python_version <" = "3.10","version >=" = "3.10.0.0"}
 black = {editable = true,extras = ["d"],path = "."}
 dataclasses = {"python_version <" = "3.7","version >" = "0.1.3"}

README.md

@@ -41,7 +41,8 @@ Try it out now using the [Black Playground](https://black.vercel.app). Watch the
 
 _Black_ can be installed by running `pip install black`. It requires Python 3.6.2+ to
 run. If you want to format Python 2 code as well, install with
-`pip install black[python2]`.
+`pip install black[python2]`. If you want to format Jupyter Notebooks, install with
+`pip install black[jupyter]`.
 
 If you can't wait for the latest _hotness_ and want to install from GitHub, use:
 

docs/faq.md

@@ -37,6 +37,29 @@ Most likely because it is ignored in `.gitignore` or excluded with configuration
 [file collection and discovery](usage_and_configuration/file_collection_and_discovery.md)
 for details.
 
+## Why is my Jupyter Notebook cell not formatted?
+
+_Black_ is timid about formatting Jupyter Notebooks. Cells containing any of the
+following will not be formatted:
+
+- automagics (e.g. `pip install black`)
+- multiline magics, e.g.:
+
+  ```python
+  %timeit f(1, \
+          2, \
+          3)
+  ```
+
+- code which `IPython`'s `TransformerManager` would transform magics into, e.g.:
+
+  ```python
+  get_ipython().system('ls')
+  ```
+
+- invalid syntax, as it can't be safely distinguished from automagics in the absense of
+  a running `IPython` kernel.
+
 ## Why are Flake8's E203 and W503 violated?
 
 Because they go against PEP 8. E203 falsely triggers on list

docs/getting_started.md

@@ -18,7 +18,8 @@ Also, you can try out _Black_ online for minimal fuss on the
 
 _Black_ can be installed by running `pip install black`. It requires Python 3.6.2+ to
 run, but can format Python 2 code too. Python 2 support needs the `typed_ast`
-dependency, which be installed with `pip install black[python2]`.
+dependency, which be installed with `pip install black[python2]`. If you want to format
+Jupyter Notebooks, install with `pip install black[jupyter]`.
 
 If you can't wait for the latest _hotness_ and want to install from GitHub, use:
 

pyproject.toml

@@ -31,6 +31,7 @@ build-backend = "setuptools.build_meta"
 optional-tests = [
   "no_python2: run when `python2` extra NOT installed",
   "no_blackd: run when `d` extra NOT installed",
+  "no_jupyter: run when `jupyter` extra NOT installed",
 ]
 markers = [
   "incompatible_with_mypyc: run when testing mypyc compiled black"

setup.py

@@ -104,14 +104,15 @@ def find_python_files(base: Path) -> List[Path]:
         "regex>=2020.1.8",
         "pathspec>=0.8.1, <1",
         "dataclasses>=0.6; python_version < '3.7'",
-        "typing_extensions>=3.7.4; python_version < '3.8'",
+        "typing_extensions>=3.10.0.0; python_version < '3.10'",
         "mypy_extensions>=0.4.3",
     ],
     extras_require={
         "d": ["aiohttp>=3.6.0", "aiohttp-cors>=0.4.0"],
         "colorama": ["colorama>=0.4.3"],
         "python2": ["typed-ast>=1.4.2"],
         "uvloop": ["uvloop>=0.15.2"],
+        "jupyter": ["ipython>=7.8.0", "tokenize-rt>=3.2.0"],
     },
     test_suite="tests.test_black",
     classifiers=[