Black failing to check or format file

[ivirshup]

Describe the bug A clear and concise description of what the bug is.

Black is failing to format a file, and telling me to open an issue. I think this is related to usage of # fmt: off, # fmt: on

To Reproduce Steps to reproduce the behavior:

Running black . --check --diff on this PR, or just black on this file

The log file

  File "/usr/local/lib/python3.7/site-packages/black.py", line 3762, in assert_equivalent
    dst_ast = parse_ast(dst)
  File "/usr/local/lib/python3.7/site-packages/black.py", line 3686, in parse_ast
    return ast27.parse(src)
  File "/usr/local/lib/python3.7/site-packages/typed_ast/ast27.py", line 50, in parse
    return _ast27.parse(source, filename, mode)
import collections.abc as cabc
from functools import singledispatch
from typing import Union, Sequence, Optional, Tuple

import numpy as np
import pandas as pd
from scipy.sparse import spmatrix


Index1D = Union[slice, int, str, np.int64, np.ndarray]
Index = Union[Index1D, Tuple[Index1D, Index1D], spmatrix]


def _normalize_indices(
    index: Optional[Index], names0: pd.Index, names1: pd.Index
) -> Tuple[slice, slice]:
    # deal with tuples of length 1
    if isinstance(index, tuple) and len(index) == 1:
        index = index[0]
    # deal with pd.Series
    if isinstance(index, pd.Series):
        index: Index = index.values
    if isinstance(index, tuple):
        if len(index) > 2:
            raise ValueError("AnnData can only be sliced in rows and columns.")
        # deal with pd.Series
        # TODO: The series should probably be aligned first
        if isinstance(index[1], pd.Series):
            index = index[0], index[1].values
        if isinstance(index[0], pd.Series):
            index = index[0].values, index[1]
    ax0, ax1 = unpack_index(index)
    ax0 = _normalize_index(ax0, names0)
    ax1 = _normalize_index(ax1, names1)
    return ax0, ax1


def _normalize_index(
    indexer: Union[
        slice,
        np.integer,
        int,
        str,
        Sequence[Union[int, np.integer]],
        np.ndarray,
        pd.Index,
    ],
    index: pd.Index,
) -> Union[slice, int, np.ndarray]:  # ndarray of int
    if not isinstance(index, pd.RangeIndex):
        assert (
            index.dtype != float and index.dtype != int
        ), "Don’t call _normalize_index with non-categorical/string names"

    # the following is insanely slow for sequences,
    # we replaced it using pandas below
    def name_idx(i):
        if isinstance(i, str):
            i = index.get_loc(i)
        return i

    if isinstance(indexer, slice):
        start = name_idx(indexer.start)
        stop = name_idx(indexer.stop)
        # string slices can only be inclusive, so +1 in that case
        if isinstance(indexer.stop, str):
            stop = None if stop is None else stop + 1
        step = indexer.step
        return slice(start, stop, step)
    elif isinstance(indexer, (np.integer, int)):
        return indexer
    elif isinstance(indexer, str):
        return index.get_loc(indexer)  # int
    elif isinstance(indexer, (Sequence, np.ndarray, pd.Index, spmatrix, np.matrix)):
        # fmt: off
        if (
            hasattr(indexer, "shape") and
            ((indexer.shape == (index.shape[0], 1)) or (indexer.shape == (1, index.shape[0])))
        ):  # fmt: on
            if isinstance(indexer, spmatrix):
                indexer = indexer.toarray()
            indexer = np.ravel(indexer)
        if not isinstance(indexer, (np.ndarray, pd.Index)):
            indexer = np.array(indexer)
        if issubclass(indexer.dtype.type, (np.integer, np.floating)):
            return indexer  # Might not work for range indexes
        elif issubclass(indexer.dtype.type, np.bool_):
            if indexer.shape != index.shape:
                raise IndexError(
                    f"Boolean index does not match AnnData’s shape along this "
                    f"dimension. Boolean index has shape {indexer.shape} while "
                    f"AnnData index has shape {index.shape}."
                )
            positions = np.where(indexer)[0]
            return positions  # np.ndarray[int]
        else:  # indexer should be string array
            positions = index.get_indexer(indexer)
            if np.any(positions < 0):
                not_found = indexer[positions < 0]
                raise KeyError(
                    f"Values {list(not_found)}, from {list(indexer)}, "
                    "are not valid obs/ var names or indices."
                )
            return positions  # np.ndarray[int]
        else:
            raise IndexError(f"Unknown indexer {indexer!r} of type {type(indexer)}")

    def unpack_index(index: Index) -> Tuple[Index1D, Index1D]:
        if not isinstance(index, tuple):
            return index, slice(None)
        elif len(index) == 2:
            return index
        elif len(index) == 1:
            return index[0], slice(None)
        else:
            raise IndexError("invalid number of indices")

    @singledispatch
    def _subset(a: Union[np.ndarray, spmatrix, pd.DataFrame], subset_idx: Index):
        # Select as combination of indexes, not coordinates
        # Correcting for indexing behaviour of np.ndarray
        if all(isinstance(x, cabc.Iterable) for x in subset_idx):
            subset_idx = np.ix_(*subset_idx)
        return a[subset_idx]

    @_subset.register(pd.DataFrame)
    def _subset_df(df: pd.DataFrame, subset_idx: Index):
        return df.iloc[subset_idx]

Expected behavior A clear and concise description of what you expected to happen.

No error.

Environment (please complete the following information):

This is occurring locally and on travis builds.

Locally, I've got:

$ black --version
black, version 19.10b0
$ python3 --version
Python 3.7.6

Does this bug also happen on master? To answer this, you have two options:

This bug does not occur using the online formatter.

[ichard26]

Please note that this bug is actually present in master on black.now.sh on commit 243230

See how the level of indentation is messed up.

Also the wrong error message is probably similar to #1263. This comment explains it somewhat well: #1263 (comment). I haven't checked if the wrong error message issue is related though.

[ichard26]

The behaviour exhibited here tells me that this is just another example of #569. If the # fmt: off comment is removed, the file formats just fine. Unfortunately you cannot easily stop Black from only formatting an if statement because # fmt: on won't work in the expected locations:

(black) ichard26@acer-ubuntu:~/programming/oss/black$ cat temp.py
# fmt: off
if ( (
    this_is_a_terrible_if_statement ) ):
    # fmt: on
    test = 'please reformat this'

# fmt: off
if ( (
    this_is_a_terrible_if_statement) ):
# fmt: on
     test = 'please reformat this'
(black) ichard26@acer-ubuntu:~/programming/oss/black$ black temp.py --diff --color --config tests/empty.toml
All done! ✨ 🍰 ✨
1 file would be left unchanged.

You have to put fmt: on after the whole if block to have Black leave the if statement be left alone. And when an indented # fmt: off applies after the end of its block, mayhem ensues (i.e. issue #569 happens).

Closing as a duplicate. Thanks for giving us another reproduction example though!