File and Path completions mostly broken in bash

[maszczyn]

First Check

• I added a very descriptive title here.
• I used the GitHub search to find a similar question and didn't find it.
• I searched the Typer documentation, with the integrated search.
• I already searched in Google "How to X in Typer" and didn't find any information.
• I already read and followed all the tutorial in the docs and didn't find an answer.
• I already checked if it is not related to Typer but to Click.

Commit to Help

• I commit to help with one of those options 👆

Example Code

#!/usr/bin/python3

# filename: test_typer_path

from pathlib import Path
from typing_extensions import Annotated

import typer

app = typer.Typer()


@app.command()
def path(the_path: Path = typer.Argument(file_okay=False, dir_okay=True, exists=True)):
    print(f"Path: {the_path}")


@app.command()
def file(text_file: Annotated[typer.FileText, typer.Argument(file_okay=True, dir_okay=False, exists=True)]):
    print(f"Text file path: {text_file.name}")


if __name__ == "__main__":
    app()

Description

The problem is with bash shell completions for files & directories in Typer (I've checked Click and it behaves as expected in that regard, that's what makes me confident it's an issue within Typer, not Click).

How to reproduce the issue

1. Install bash completions for the script from "Example code":

eval "$(test_typer_path --show-completion bash)"

2. Make sure that Typer's command completions are working

test_typer_path [TAB][TAB]    # gives suggestions as expected: file  path
test_typer_path p[TAB]        # single [tab] completes unambiguous command: "tp path"

3. Attempt DIR path completion (the example code provided above should result in suggesting directory names only)

Actual result

test_typer_path path [TAB]
  # the shell command stays intact, despite there is an unambiguous part of 
  # dirnames that should be auto-completed here (both directories start with an "a")

test_typer_path path [TAB][TAB]
  # all directories and files are suggested:
  # a_directory/  another_dir/  another_file  some_file

test_typer_path path a[TAB][TAB]
  # "a" is interpreted as the final argument value, and a space gets added to separate it from further arguments / options:
  # "test_typer_path path a " 
  # (with a trailing space)

Expected result

test_typer_path path [TAB]
  # shell command should get auto-completed with the unambiguous part
  # "test_typer_path path a"
  # (notice there's no space at the end of auto-completion)

test_typer_path path a[TAB][TAB]
  # only directory names should be suggested here:
  # a_directory/ another_dir/

The expected result is in line with Click's behavior. I used the following Click code to check it:

#!/usr/bin/python3

# filename: test_click_path

import click


@click.group()
def test_click_path():
    pass


@test_click_path.command()
@click.argument('path', nargs=1, type=click.Path(dir_okay=True, file_okay=False))
def arg(path):
    print(path)


@test_click_path.command()
@click.option('--path', type=click.Path(), help='test option path complete')
def opt(path):
    print(path)


if __name__ == '__main__':
    test_click_path()

The most simplistic way to install bash completions for the above script in current shell session is this:

eval "$(_TEST_CLICK_PATH_COMPLETE=bash_source test_click_path)"

I think this issue also impacted another user as described in this discussion.

Possible root cause - a lead to follow

I'm no shell expert, so the "lead" I will point here might be totally wrong.

I noticed the the shell completion scripts generated by Typer and Click differ a lot.

Typer:

_test_typer_path_completion() {
    local IFS=$'
'
    COMPREPLY=( $( env COMP_WORDS="${COMP_WORDS[*]}" \
                   COMP_CWORD=$COMP_CWORD \
                   _TEST_TYPER_PATH_COMPLETE=complete_bash $1 ) )
    return 0
}

complete -o default -F _test_typer_path_completion test_typer_path

Click:

_test_click_path_completion() {
    local IFS=$'\n'
    local response

    response=$(env COMP_WORDS="${COMP_WORDS[*]}" COMP_CWORD=$COMP_CWORD _TEST_CLICK_PATH_COMPLETE=bash_complete $1)

    for completion in $response; do
        IFS=',' read type value <<< "$completion"

        if [[ $type == 'dir' ]]; then
            COMPREPLY=()
            compopt -o dirnames
        elif [[ $type == 'file' ]]; then
            COMPREPLY=()
            compopt -o default
        elif [[ $type == 'plain' ]]; then
            COMPREPLY+=($value)
        fi
    done

    return 0
}

_test_click_path_completion_setup() {
    complete -o nosort -F _test_click_path_completion test_click_path
}

_test_click_path_completion_setup;

There are two things that grab my attention, but I am not sure if they are relevant in this context:

1. Click has some logic to differentiate files and dirs, while Typer has none
2. Click uses complete -o nosort, while Typer uses complete -o default

Operating System

Linux

Operating System Details

No response

Typer Version

0.9.0

Python Version

Python 3.10.12

Additional Context

Click version: 8.1.7

[i30817]

Sorry for dumping a lot of code here, I'm on a tablet. This is what I did for myself

def autocomplete(ctx: Context, args: List[str], incomplete: str): 
     conf = Path(ctx.args[0]) if ctx.args else CONFIG 
     if not conf or not conf.is_file() or not os.access(conf, os.R_OK): 
         return [] 
     playlist_dir = getPath(conf, "playlist_directory", "playlists") 
     if not playlist_dir or not playlist_dir.is_dir() or not os.access(playlist_dir, os.R_OK): 
         return [] 
     # if the name doesn't start with incomplete this is empty (like passing ./) 
     # if it returns a empty list then the program falls back to the cmdline parsing 
     return [ 
         pl.stem 
         for pl in playlist_dir.glob("./*.lpl") 
         if pl.is_file() and os.access(pl, os.R_OK) and pl.name.startswith(incomplete) 
     ] 
  
  
 # this is necessary because typer has a bug 
 # where any path typed is considered 'complete' 
 # for arguments (even if exists=True) and a space 
 # is added to the end - this works around it. 
 def complete_path(): 
     return [] 
  
  
 ##################### 
 # Main programs code 
 ##################### 
  
  
 def mainfuzzsingle( 
     cfg: Path = Argument( 
         CONFIG, 
         help="Path to the retroarch cfg file. If not default, asked from the user.", 
         exists=True, 
         file_okay=True, 
         dir_okay=False, 
         readable=True, 
         resolve_path=True, 
         autocompletion=complete_path, 
     ), 
     playlist: str = Option( 
         None, 
         metavar="NAME", 
         shell_complete=autocomplete, 
         help="Playlist name with labels used for thumbnail fuzzy matching. If not provided, asked from the user.", 
     ),

So everything but files are handled normally, argument files use autocompletion callback which falls back to console, and only autocomplete paths with spaces if you open a quote at the start (not ideal).

I think I also need quotes in the option arguments because I didn't bother listing the console quoted space version as a possibility and it looks better but I'm not sure (in a tablet as I write). If they aren't of a restricted set I expect, they're passed to the console and treated as a console file.

I don't recall but I think I used different callbacks because they are subtly different in behavior in spite of the documentation saying they're the same.

[maszczyn]

I've been thinking to get myself a similar workaround. Thank you for sharing yours, @i30817 !

[i30817]

Its neat to be able to workaround this and specialize some auto complete options to do cool things based on previous data, but the slowness kind of kills me. If the autocomplete feature was nice enough to only slowdown the options it is going to affect it would be if not fine, expected. But it appears to everything afaict.

[maszczyn]

I came up with the following work-around for the broken shell completion:

def path_autocomplete(
        file_okay: bool = True,
        dir_okay: bool = True,
        writable: bool = False,
        readable: bool = True,
        allow_dash: bool = False,
        match_wildcard: Optional[str] = None,
) -> Callable[[str], list[str]]:
    def wildcard_match(string: str, pattern: str) -> bool:
        regex = re.escape(pattern).replace(r'\?', '.').replace(r'\*', '.*')
        return re.fullmatch(regex, string) is not None

    def completer(incomplete: str) -> list[str]:
        items = os.listdir()
        completions = []
        for item in items:
            if not file_okay and os.path.isfile(item):
                continue
            elif not dir_okay and os.path.isdir(item):
                continue

            if readable and not os.access(item, os.R_OK):
                continue
            if writable and not os.access(item, os.W_OK):
                continue

            completions.append(item)

        if allow_dash:
            completions.append('-')

        if match_wildcard is not None:
            completions = filter(lambda i: wildcard_match(i, match_wildcard), completions)

        return [i for i in completions if i.startswith(incomplete)]

    return completer

It contains a subset to arguments normally provided to Argument and Option to mimic the desired behavior, plus one additional (optional) filter: match_wildcard accepting * and ?.

You can use it like this:

@app.command()
def some_command(filename: Annotated[Path, typer.Argument(autocompletion=path_autocomplete(file_okay=True, dir_okay=False, readable=True, match_wildcard="foo*"))]):
    # notice the file_okay and the rest of path-related arguments goes to path_autocomplete, to Argument
    # this makes the code easy to cleanup when the bug is fixed
    pass

[RoketFlame]

Same problem with zsh and oh-my-zsh. How to represent.

Example from official doc:

from pathlib import Path
from typing import Optional

import typer
from typing_extensions import Annotated


def main(config: Annotated[Optional[Path], typer.Option()] = None):
    if config is None:
        print("No config file")
        raise typer.Abort()
    if config.is_file():
        text = config.read_text()
        print(f"Config file contents: {text}")
    elif config.is_dir():
        print("Config is a directory, will use all its config files")
    elif not config.exists():
        print("The config doesn't exist")


if __name__ == "__main__":
    typer.run(main)

Actual result

typer ./main.py --config [TAB]
# No hints appear, only a single space is added to the end of the entered command
# If you continue to press [TAB], more spaces will be added

Expected result

typer ./main.py --config [TAB]
# Hints according to Path type