-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use spack commands --format=bash
to generate shell completion
#14393
Conversation
ccffede
to
bc1190a
Compare
Thanks, I'll take a look. It looks like fish completions are defined somewhat differently (but still tediously). I will first try existing bash-to-fish completions generators. Otherwise I can write a I agree that we don't want to maintain 2 parallel 1000+ line scripts, and this looks like the solution. |
@tgamblin This looks great! Some thoughts:
If you want to focus on the concretizer, I would be happy to hack on this over the next couple weeks. |
Sounds fine to me. I'd call the completion formats
Yep this is what I meant by "smarts around things like completing package names".
Or just have this command prepend the initial function definitions to the generated output -- that's what I was thinking based on how the bash completion currently looks. Up to you.
This is exactly what I was thinking. basically if the argument name is
Yes please! |
Question for everyone involved: Previously, the tab-completion for something like Disadvantage:
Advantage:
I think this situation isn't particularly controversial, just because commands are added/removed so infrequently. But what about package names? Currently, every time you hit Vote 👍 for faster hard-coding and 👎 for slower but more up-to-date |
@adamjstewart The slower, up-to-date methods for command and package completion will include extension commands and packages from configured external repos, respectively. I could go with caching based on ages of configuration files and the modification times on configured |
@chissg Correct, hard-coding wouldn't work for commands/packages that only exist in other repositories. Commands like If anyone can figure out how to cache these functions from within Bash, that would make things a lot faster. |
Okay, I figured out how to cache the results in Bash, I think this will make everyone happy. Basically, you can do something like this: diff --git a/share/spack/spack-completion.bash b/share/spack/spack-completion.bash
index e1b10f676d..c05b146363 100755
--- a/share/spack/spack-completion.bash
+++ b/share/spack/spack-completion.bash
@@ -87,7 +87,8 @@ function _bash_completion_spack {
# Make sure function exists before calling it
if [[ "$(type -t $subfunction)" == "function" ]]
then
- COMPREPLY=($($subfunction))
+ $subfunction
+ COMPREPLY=($(compgen -W "$SPACK_COMPREPLY" -- "$cur"))
fi
}
@@ -718,16 +719,16 @@ function _spack_info {
function _spack_install {
if $list_options
then
- compgen -W "-h --help --only -u --until -j --jobs --overwrite
+ SPACK_COMPREPLY="-h --help --only -u --until -j --jobs --overwrite
--keep-prefix --keep-stage --dont-restage --use-cache
--no-cache --cache-only --show-log-on-error --source
-n --no-checksum -v --verbose --fake --only-concrete
-f --file --clean --dirty --test --run-tests
--log-format --log-file --help-cdash -y --yes-to-all
--cdash-upload-url --cdash-build --cdash-site
- --cdash-track --cdash-buildstamp" -- "$cur"
+ --cdash-track --cdash-buildstamp"
else
- compgen -W "$(_all_packages)" -- "$cur"
+ _all_packages
fi
}
@@ -1347,7 +1348,11 @@ function _subcommands {
}
function _all_packages {
- spack list
+ if [[ -z $SPACK_ALL_PACKAGES ]]
+ then
+ SPACK_ALL_PACKAGES=$(spack list)
+ fi
+ SPACK_COMPREPLY=$SPACK_ALL_PACKAGES
}
function _all_resource_hashes { Caching wasn't working before because we were running each function in a subshell, and you can't pass variables from a subshell to a supershell. By running everything in the current shell, we can cache the results of each call to |
bc1190a
to
0dc19c6
Compare
For the record, if we want to unit test the Bash completion script, we can do something like https://brbsix.github.io/2015/11/29/accessing-tab-completion-programmatically-in-bash/ |
spack completion
command to generate shell completionspack commands --format=bash
to generate shell completion
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Still need to replace positionals with calls to subfunctions, but this is pretty close now!
str: the function definition beginning | ||
""" | ||
name = prog.replace('-', '_').replace(' ', '_') | ||
return '_{0} () {{'.format(name) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I went with the POSIX-compliant function definition so this won't need to be changed for most shells.
@@ -0,0 +1,194 @@ | |||
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How do we get GitHub/vim to recognize this filetype, add a shebang or use .gitattributes
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
.gitattributes
works for GitHub but not for vim. # vim: set filetype=sh :
didn't work for me locally, but maybe I don't have modelines enabled. A shebang worked, but since this script is sourced, I'm not sure if it should have a shebang.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could also replace the extension with .bash
or .sh
, but then it isn't clear what the difference is between this file and the actual file you source.
done | ||
} | ||
|
||
complete -o bashdefault -o default -F _bash_completion_spack spack |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Git's tab completion also uses -o bashdefault
which we weren't using before. I tested this with the system Bash on macOS (3.2.57) and Spack-installed Bash (5.0.11).
How do we want to handle the translation of positional args to Bash subroutines? I see three possible avenues:
|
@adamjstewart I'm in favor of 1, specifically: |
9926e86
to
ed55a9f
Compare
I also like (1), except this part:
Some commands can take multiple specs/package names/etc., and some can only take one. I think that should be understandable from the command line options, and if we're consistent about it users will have an easier time understanding the different commands. It doesn't have to be relevant for completion, though -- the completion can just know that both of, e.g., |
EDIT: Now I remember why |
contains 'python' _spack_completions spack extensions '' | ||
|
||
# XFAIL: Fails for Python 2.6 because pkg_resources not found? | ||
#contains 'compilers.py' _spack_completions spack test '' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is likely due to #9034
How can I see coverage for EDIT: nvm, found it: https://codecov.io/gh/spack/spack/src/features%2Fautogenerate-completion/share/spack/spack-completion.bash Any idea why it doesn't consider function definitions to be covered? EDIT: This was likely fixed by SimonKagstrom/kcov#258. The version of Any idea why that one if-statement claims not to be covered? EDIT: Putting things on a single line gets us to 100% coverage! |
Alright, we now cache all subroutine calls in our tab completion. This makes repeated tab completion of the same command, like Unfortunately, the |
set -u | ||
|
||
# Source setup-env.sh before tests | ||
. share/spack/setup-env.sh | ||
. "$SHARE_DIR/setup-env.sh" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Previously this script only worked if you ran it from $SPACK_ROOT
_config_sections() { | ||
if [[ -z "${SPACK_CONFIG_SECTIONS:-}" ]] | ||
then | ||
SPACK_CONFIG_SECTIONS="compilers mirrors repos packages modules config upstreams" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See #14474
_extensions() { | ||
if [[ -z "${SPACK_EXTENSIONS:-}" ]] | ||
then | ||
SPACK_EXTENSIONS="aspell go-bootstrap go icedtea jdk kim-api lua matlab mofem-cephas octave openjdk perl python r ruby rust tcl yorick" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See #14473
else | ||
compgen -W "regenerate enable disable" -- "$cur" | ||
SPACK_COMPREPLY="" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is something I still need to fix. The actual positional arg is {regenerate,enable,disable}
. If we reimplement this using subcommands, the problem goes away.
else | ||
compgen -W "$(installed_packages)" -- "$cur" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was a bug, we were missing the leading underscore. Same below for _spack_gpg_sign
.
@tgamblin There are still a few bugs to work out, but this PR should be ready for review. Note that the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@adamjstewart: a couple minor change requests and I think this is ready to merge.
|
||
def format(self, prog, description, usage, | ||
positionals, optionals, subcommands): | ||
string = self.begin_command(prog) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be a StringIO
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's the benefit of using StringIO
? I've never used it before.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See https://docs.python.org/3/faq/programming.html#what-is-the-most-efficient-way-to-concatenate-many-strings-together. When writing a large file like this use string IO for efficiency.
usage (str): the command usage | ||
positionals (list): list of positional arguments | ||
optionals (list): list of optional arguments | ||
subcommands (list): list of subcommand parsers |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These 6 args are kind of begging to be a little ParsedArgs
data class so that we don't have to pass all 6 around.
@adamjstewart: when you push this, can you squash the commits into a single one (or a few) by you that I can rebase? If I squash this PR, it'll credit the work to me (as I opened the bare-bones prototype) |
This PR adds a `--format=bash` option to `spack commands` to auto-generate the Bash programmable tab completion script. It can be extended to work for other shells. Progress: - [x] Fix bug in superclass initialization in `ArgparseWriter` - [x] Refactor `ArgparseWriter` (see below) - [x] Ensure that output of old `--format` options remains the same - [x] Add `ArgparseCompletionWriter` and `BashCompletionWriter` - [x] Add `--aliases` option to add command aliases - [x] Standardize positional argument names - [x] Tests for `spack commands --format=bash` coverage - [x] Tests to make sure `spack-completion.bash` stays up-to-date - [x] Tests for `spack-completion.bash` coverage - [x] Speed up `spack-completion.bash` by caching subroutine calls This PR also necessitates a significant refactoring of `ArgparseWriter`. Previously, `ArgparseWriter` was mostly a single `_write` method which handled everything from extracting the information we care about from the parser to formatting the output. Now, `_write` only handles recursion, while the information extraction is split into a separate `parse` method, and the formatting is handled by `format`. This allows subclasses to completely redefine how the format will appear without overriding all of `_write`. Co-Authored-by: Todd Gamblin <tgamblin@llnl.gov>
572773f
to
48f5abd
Compare
This PR adds a `--format=bash` option to `spack commands` to auto-generate the Bash programmable tab completion script. It can be extended to work for other shells. Progress: - [x] Fix bug in superclass initialization in `ArgparseWriter` - [x] Refactor `ArgparseWriter` (see below) - [x] Ensure that output of old `--format` options remains the same - [x] Add `ArgparseCompletionWriter` and `BashCompletionWriter` - [x] Add `--aliases` option to add command aliases - [x] Standardize positional argument names - [x] Tests for `spack commands --format=bash` coverage - [x] Tests to make sure `spack-completion.bash` stays up-to-date - [x] Tests for `spack-completion.bash` coverage - [x] Speed up `spack-completion.bash` by caching subroutine calls This PR also necessitates a significant refactoring of `ArgparseWriter`. Previously, `ArgparseWriter` was mostly a single `_write` method which handled everything from extracting the information we care about from the parser to formatting the output. Now, `_write` only handles recursion, while the information extraction is split into a separate `parse` method, and the formatting is handled by `format`. This allows subclasses to completely redefine how the format will appear without overriding all of `_write`. Co-Authored-by: Todd Gamblin <tgamblin@llnl.gov>
48f5abd
to
b989e52
Compare
This PR adds a `--format=bash` option to `spack commands` to auto-generate the Bash programmable tab completion script. It can be extended to work for other shells. Progress: - [x] Fix bug in superclass initialization in `ArgparseWriter` - [x] Refactor `ArgparseWriter` (see below) - [x] Ensure that output of old `--format` options remains the same - [x] Add `ArgparseCompletionWriter` and `BashCompletionWriter` - [x] Add `--aliases` option to add command aliases - [x] Standardize positional argument names - [x] Tests for `spack commands --format=bash` coverage - [x] Tests to make sure `spack-completion.bash` stays up-to-date - [x] Tests for `spack-completion.bash` coverage - [x] Speed up `spack-completion.bash` by caching subroutine calls This PR also necessitates a significant refactoring of `ArgparseWriter`. Previously, `ArgparseWriter` was mostly a single `_write` method which handled everything from extracting the information we care about from the parser to formatting the output. Now, `_write` only handles recursion, while the information extraction is split into a separate `parse` method, and the formatting is handled by `format`. This allows subclasses to completely redefine how the format will appear without overriding all of `_write`. Co-Authored-by: Todd Gamblin <tgamblin@llnl.gov>
b989e52
to
bc438ee
Compare
This PR broke completion of flags:
|
fixes #8780
This PR adds a
--format=bash
option tospack commands
to auto-generate the Bash programmable tab completion script. It can be extended to work for other shells.Progress:
ArgparseWriter
ArgparseWriter
(see below)--format
options remains the sameArgparseCompletionWriter
andBashCompletionWriter
--aliases
option to add command aliasesspack commands --format=bash
coveragespack-completion.bash
stays up-to-datespack-completion.bash
coveragespack-completion.bash
by caching subroutine callsThings I would like to save for future PRs:
spack commands --format=rst | head
in Python 3 (see comment)This PR also necessitates a significant refactoring of
ArgparseWriter
. Previously,ArgparseWriter
was mostly a single_write
method which handled everything from extracting the information we care about from the parser to formatting the output. Now,_write
only handles recursion, while the information extraction is split into a separateparse
method, and the formatting is handled byformat
. This allows subclasses to completely redefine how the format will appear without overriding all of_write
.