Build failure: python311Packages.click

[RuRo]

Steps To Reproduce

Steps to reproduce the behavior:

1. have a filesystem that doesn't accept non-utf8 characters
2. build python-click (i.e. nix build nixpkgs#python311Packages.click --rebuild)

Build log

=================================== FAILURES ===================================
_________________________ test_file_surrogates[type0] __________________________

self = <click.types.File object at 0xf63ffab0>
value = PosixPath('/build/pytest-of-nixbld/pytest-0/test_file_surrogates_type0_0/\udcff')
param = None, ctx = None

    def convert(
        self,
        value: t.Union[str, "os.PathLike[str]", t.IO[t.Any]],
        param: t.Optional["Parameter"],
        ctx: t.Optional["Context"],
    ) -> t.IO[t.Any]:
        if _is_file_like(value):
            return value
    
        value = t.cast("t.Union[str, os.PathLike[str]]", value)
    
        try:
            lazy = self.resolve_lazy_flag(value)
    
            if lazy:
                lf = LazyFile(
                    value, self.mode, self.encoding, self.errors, atomic=self.atomic
                )
    
                if ctx is not None:
                    ctx.call_on_close(lf.close_intelligently)
    
                return t.cast(t.IO[t.Any], lf)
    
>           f, should_close = open_stream(
                value, self.mode, self.encoding, self.errors, atomic=self.atomic
            )

/nix/store/y9dhvsz01b8n3bccpgwk6kfhhgglblfs-python3.11-click-8.1.7/lib/python3.11/site-packages/click/types.py:724: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/nix/store/y9dhvsz01b8n3bccpgwk6kfhhgglblfs-python3.11-click-8.1.7/lib/python3.11/site-packages/click/_compat.py:396: in open_stream
    return _wrap_io_open(filename, mode, encoding, errors), True
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

file = '/build/pytest-of-nixbld/pytest-0/test_file_surrogates_type0_0/\udcff'
mode = 'r', encoding = None, errors = 'strict'

    def _wrap_io_open(
        file: t.Union[str, "os.PathLike[str]", int],
        mode: str,
        encoding: t.Optional[str],
        errors: t.Optional[str],
    ) -> t.IO[t.Any]:
        """Handles not passing ``encoding`` and ``errors`` in binary mode."""
        if "b" in mode:
            return open(file, mode)
    
>       return open(file, mode, encoding=encoding, errors=errors)
E       OSError: [Errno 84] Invalid or incomplete multibyte or wide character: '/build/pytest-of-nixbld/pytest-0/test_file_surrogates_type0_0/\udcff'

/nix/store/y9dhvsz01b8n3bccpgwk6kfhhgglblfs-python3.11-click-8.1.7/lib/python3.11/site-packages/click/_compat.py:370: OSError

During handling of the above exception, another exception occurred:

type = <click.types.File object at 0xf63ffab0>
tmp_path = PosixPath('/build/pytest-of-nixbld/pytest-0/test_file_surrogates_type0_0')

    @pytest.mark.parametrize(
        "type",
        [
            click.File(mode="r"),
            click.File(mode="r", lazy=True),
        ],
    )
    def test_file_surrogates(type, tmp_path):
        path = tmp_path / "\udcff"
    
        with pytest.raises(click.BadParameter, match="�': No such file or directory"):
>           type.convert(path, None, None)

tests/test_types.py:229: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/nix/store/y9dhvsz01b8n3bccpgwk6kfhhgglblfs-python3.11-click-8.1.7/lib/python3.11/site-packages/click/types.py:741: in convert
    self.fail(f"'{format_filename(value)}': {e.strerror}", param, ctx)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <click.types.File object at 0xf63ffab0>
message = "'/build/pytest-of-nixbld/pytest-0/test_file_surrogates_type0_0/�': Invalid or incomplete multibyte or wide character"
param = None, ctx = None

    def fail(
        self,
        message: str,
        param: t.Optional["Parameter"] = None,
        ctx: t.Optional["Context"] = None,
    ) -> "t.NoReturn":
        """Helper method to fail with an invalid value message."""
>       raise BadParameter(message, ctx=ctx, param=param)
E       click.exceptions.BadParameter: '/build/pytest-of-nixbld/pytest-0/test_file_surrogates_type0_0/�': Invalid or incomplete multibyte or wide character

/nix/store/y9dhvsz01b8n3bccpgwk6kfhhgglblfs-python3.11-click-8.1.7/lib/python3.11/site-packages/click/types.py:136: BadParameter

During handling of the above exception, another exception occurred:

type = <click.types.File object at 0xf63ffab0>
tmp_path = PosixPath('/build/pytest-of-nixbld/pytest-0/test_file_surrogates_type0_0')

    @pytest.mark.parametrize(
        "type",
        [
            click.File(mode="r"),
            click.File(mode="r", lazy=True),
        ],
    )
    def test_file_surrogates(type, tmp_path):
        path = tmp_path / "\udcff"
    
>       with pytest.raises(click.BadParameter, match="�': No such file or directory"):
E       AssertionError: Regex pattern did not match.
E        Regex: "�': No such file or directory"
E        Input: "'/build/pytest-of-nixbld/pytest-0/test_file_surrogates_type0_0/�': Invalid or incomplete multibyte or wide character"

tests/test_types.py:228: AssertionError
_________________________ test_file_surrogates[type1] __________________________

self = <click.types.File object at 0xf63ffa30>
value = PosixPath('/build/pytest-of-nixbld/pytest-0/test_file_surrogates_type1_0/\udcff')
param = None, ctx = None

    def convert(
        self,
        value: t.Union[str, "os.PathLike[str]", t.IO[t.Any]],
        param: t.Optional["Parameter"],
        ctx: t.Optional["Context"],
    ) -> t.IO[t.Any]:
        if _is_file_like(value):
            return value
    
        value = t.cast("t.Union[str, os.PathLike[str]]", value)
    
        try:
            lazy = self.resolve_lazy_flag(value)
    
            if lazy:
>               lf = LazyFile(
                    value, self.mode, self.encoding, self.errors, atomic=self.atomic
                )

/nix/store/y9dhvsz01b8n3bccpgwk6kfhhgglblfs-python3.11-click-8.1.7/lib/python3.11/site-packages/click/types.py:715: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <[RecursionError('maximum recursion depth exceeded') raised in repr()] LazyFile object at 0xf63b0c30>
filename = PosixPath('/build/pytest-of-nixbld/pytest-0/test_file_surrogates_type1_0/\udcff')
mode = 'r', encoding = None, errors = 'strict', atomic = False

    def __init__(
        self,
        filename: t.Union[str, "os.PathLike[str]"],
        mode: str = "r",
        encoding: t.Optional[str] = None,
        errors: t.Optional[str] = "strict",
        atomic: bool = False,
    ):
        self.name: str = os.fspath(filename)
        self.mode = mode
        self.encoding = encoding
        self.errors = errors
        self.atomic = atomic
        self._f: t.Optional[t.IO[t.Any]]
        self.should_close: bool
    
        if self.name == "-":
            self._f, self.should_close = open_stream(filename, mode, encoding, errors)
        else:
            if "r" in mode:
                # Open and close the file in case we're opening it for
                # reading so that we can catch at least some errors in
                # some cases early.
>               open(filename, mode).close()
E               OSError: [Errno 84] Invalid or incomplete multibyte or wide character: '/build/pytest-of-nixbld/pytest-0/test_file_surrogates_type1_0/\udcff'

/nix/store/y9dhvsz01b8n3bccpgwk6kfhhgglblfs-python3.11-click-8.1.7/lib/python3.11/site-packages/click/utils.py:136: OSError

During handling of the above exception, another exception occurred:

type = <click.types.File object at 0xf63ffa30>
tmp_path = PosixPath('/build/pytest-of-nixbld/pytest-0/test_file_surrogates_type1_0')

    @pytest.mark.parametrize(
        "type",
        [
            click.File(mode="r"),
            click.File(mode="r", lazy=True),
        ],
    )
    def test_file_surrogates(type, tmp_path):
        path = tmp_path / "\udcff"
    
        with pytest.raises(click.BadParameter, match="�': No such file or directory"):
>           type.convert(path, None, None)

tests/test_types.py:229: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/nix/store/y9dhvsz01b8n3bccpgwk6kfhhgglblfs-python3.11-click-8.1.7/lib/python3.11/site-packages/click/types.py:741: in convert
    self.fail(f"'{format_filename(value)}': {e.strerror}", param, ctx)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <click.types.File object at 0xf63ffa30>
message = "'/build/pytest-of-nixbld/pytest-0/test_file_surrogates_type1_0/�': Invalid or incomplete multibyte or wide character"
param = None, ctx = None

    def fail(
        self,
        message: str,
        param: t.Optional["Parameter"] = None,
        ctx: t.Optional["Context"] = None,
    ) -> "t.NoReturn":
        """Helper method to fail with an invalid value message."""
>       raise BadParameter(message, ctx=ctx, param=param)
E       click.exceptions.BadParameter: '/build/pytest-of-nixbld/pytest-0/test_file_surrogates_type1_0/�': Invalid or incomplete multibyte or wide character

/nix/store/y9dhvsz01b8n3bccpgwk6kfhhgglblfs-python3.11-click-8.1.7/lib/python3.11/site-packages/click/types.py:136: BadParameter

During handling of the above exception, another exception occurred:

type = <click.types.File object at 0xf63ffa30>
tmp_path = PosixPath('/build/pytest-of-nixbld/pytest-0/test_file_surrogates_type1_0')

    @pytest.mark.parametrize(
        "type",
        [
            click.File(mode="r"),
            click.File(mode="r", lazy=True),
        ],
    )
    def test_file_surrogates(type, tmp_path):
        path = tmp_path / "\udcff"
    
>       with pytest.raises(click.BadParameter, match="�': No such file or directory"):
E       AssertionError: Regex pattern did not match.
E        Regex: "�': No such file or directory"
E        Input: "'/build/pytest-of-nixbld/pytest-0/test_file_surrogates_type1_0/�': Invalid or incomplete multibyte or wide character"

tests/test_types.py:228: AssertionError
=========================== short test summary info ============================
FAILED tests/test_types.py::test_file_surrogates[type0] - AssertionError: Regex pattern did not match.
FAILED tests/test_types.py::test_file_surrogates[type1] - AssertionError: Regex pattern did not match.
============= 2 failed, 586 passed, 22 skipped, 1 xfailed in 0.97s =============

Additional context

I think that this is a known issue with ZFS and darwin. There is precedent for this kind of issue in #205780 + #205846, so the solution is probably just to disable the tests/test_types.py::test_file_surrogates tests.

Edit: I have verified that adding test_file_surrogates to disabledTests fixed the issue.

Notify maintainers

@NickCao

Priorities

Add a 👍 reaction to issues you find important.

[RuRo]

Alternatively, this could/should be fixed upstream. I opened an issue pallets/click#2634 asking for that.

[mweinelt]

This is likely due to a setting like formD normalization on the relevant pool. Can't recommend enabling any normalization when building on ZFS.

[NickCao]

Is this darwin specific? Or shall we disable the test unconditionally.

[RuRo]

Is this darwin specific? Or shall we disable the test unconditionally.

It is not Darwin specific. It is filesystem specific. Personally, I ran into this issue on NixOS on ZFS (with utf8only). So either disable the test unconditionally, or wait for the upstream to fix the test.

[NickCao]

Fixed in 5bd94e6