Completion support for Nushell

[ayax79]

Completion support for Nushell. Besides the tests included, I have also tested against minikube.

[CLAassistant]

All committers have signed the CLA.

[github-actions]

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

[marckhouzam]

Hey @ayax79 this exciting! I have never used nushell but I have now installed it on my Mac.

I see that this PR generates a script containing the completions. However, this is no longer the approach used by Cobra. Instead, the program, through Cobra, provides the completions itself through the hidden __complete command. The nushell script should then call prog __complete <all other args on the command-line> to obtain the list of completions and it should use them as appropriate for nushell. You can test out the __complete command manually by doing things like prog __complete '' or prog __complete -- and such.

This approach means that we have centralized all the logic to generate completions and coded it in Go (in Cobra's completions.go file). Then Cobra generates a shell script for each supported shell which uses the __complete command to get the completions.

Following this approach would be your first step. You can look at the completion scripts for any of the other shells to see how this is done. These scripts are in files named for the shell they support.

Eventually, the nushell script should also support the ShellCompDirectives, which you will notice in the other scripts.

Feel free to ask for more info if needed.

[ayax79]

Hey @marckhouzam,

Thanks for looking at this and thanks for the feedback!

Nushell completions quite a bit differently than completions in other shells I have used (zsh). nushell_completions.go doesn't generate a script for the completions, it generates something more akin to a header file. each "export extern " maps to an external, non-nushell command. An example would be export extern "minikube start" It is telling nushell about the 'minikube start' sub-command and describing the parameters it accepts.

For example, see the minikube-completions.nu I generated.
minikube-completions.nu.txt

You can load it into nushell via (added .txt to get it to upload):
use minikube-completions.nu.txt *

I'll dig into the new model, but I would love any implementation advice.

[marckhouzam]

We had a similar situation with Fish where normal completions are a long list of choices. So in our case we have one generic completion entry which calls a function that calls prog __complete ... and processes the results.

The nushell doc talks about custom completions which seem like an avenue to investigate. For example it shows:

export extern "git push" [
    remote?: string@"nu-complete git remotes"
...

which calls git remotes to get the list of completions.

[marckhouzam]

An important advantage of Cobra's approach is that using the __complete command in your nushell script will automatically support dynamic completions provided by the program using Cobra (which your solution doesn't). Please see details here: https://github.com/spf13/cobra/blob/main/shell_completions.md#dynamic-completion-of-nouns

[ayax79]

@rsteube the author of (carapace)[https://github.com/rsteube/carapace] chimed in with some good feedback on the nushell discord that I will include too.
As already commented in the issue it is better to take the external_completer (https://www.nushell.sh/blog/2022-09-06-nushell-0_68.html#external-completions-experimental-herlon214-rsteube) approach for this. You can grab some insights from https://github.com/rsteube/carapace/blob/master/internal/shell/nushell/action.go if it's any help. For the meantime i got a custom completer for minikube and kubectl at https://github.com/rsteube/carapace-bin . Completion of cobra based commands can also be bridged if those don't work well for you: https://rsteube.github.io/carapace-bin/spec/bride.html .

[ayax79]

I have played around with a few options and haven't found anything that works well utilizing __command.

This works for only the first level for commands and breaks flags:

def 'nu-complete minikube' [] {
	minikube __command
}

export extern minikube [
	command?: string@'nu-complete minikube'
]

Broken help flag:

To access a flag you have to bypass nushell via:

^minikube --help

By adding another level:

def 'nu-complete minikube start' [] {
    minikube start __command
}

export extern 'minikube start' [
	command?: string@'nu-complete minikube start'
]

This overrides the completion list and removes the description (unless it is manually added as comment). It also breaks any flags.

The other option is to utilize the external completer support. This however only provides an option for one global fallback completer. This how carapace-bin works with nushell.

I'll poke around more, but it seems like I am may be running out of options.

[marckhouzam]

Thanks for continuing the investigation @ayax79.

In the above comment you typed __command everywhere, but I assume you meant __complete?
Also be aware that the format is prog __complete <other args>, so the __complete command should always be the first argument to the program.

It is also important to pass the current command-line arguments that the user typed. So, if the user typed minikube start --<tab> to complete flags, the way to get the completions is to call minikube __complete start --.

Finally, if the last character on the command-line before the user pressed TAB is a space, you must pass an extra empty argument. So, if the user typed minikube start <TAB> (notice the space after start) you need to call minikube __complete start "". This complexity allows cobra to tell the difference between the user typing minikube start<TAB> which is request to complete the start argument, versus minikube start <TAB> which is a request to complete the argument after start.

You can find documentation about all that here: https://github.com/spf13/cobra/blob/main/shell_completions.md#debugging

[rsteube]

Yeah you'll definitely have no success with the extern approach, it won't allow unknown flags.
Sorry if it wasn't clear before, but you can do some nu scripting magic to support mutliple completers - it's just bad in regards to "registering" them.

let external_completer = {|spans| 
  {
    $spans.0: { } # default
    kubectl: { kubectl __complete ($spans | into df | slice 1 1000) | handleOutput }
    minikube: { minikube __complete ($spans | into df | slice 1 1000) | handleOutput }
  } | get $spans.0 | each {|it| do $it}
}

[ayax79]

I finally got some more time to work on this. I also spent some time digging into the cobra completions and the nushell external completions code .

This seems to work, although it needs some more testing. There might be some weirdness around mixing flags and commands:

# Executes a cobra apps __complete cmd with the following span
export def exec_cobra_complete [
  cmd: string,  # The cobra command (e.g. kubectl)
  spans: list,  # The list of spans
 ] {
  # skip the first entry in the span (the command) and join the rest of the span to create __complete args
  let cmd_args = ($spans | skip 1 | str join ' ') 

  # If the last span entry was empty add "" to the end of the command args
  let cmd_args = if ($spans | last | str trim | is-empty) {
    $'($cmd_args) ""'
  } else {
    $cmd_args
  }

  # The full command to be executed
  let full_cmd = $'($cmd) __complete ($cmd_args)'

  # Since nushell doesn't have anything like eval, execute in a subshell
  let result = (do -i { nu -c $"'($full_cmd)'" } | complete)

  # Create a record with all completion related info. 
  # directive and directive_str are for posterity
  let stdout_lines = ($result.stdout | lines)
  {
    completions: ($stdout_lines | drop | lines | parse "{value}\t{description}")
    directive_str: ($result.stderr)
    directive: ($stdout_lines | last)
  }
}

# Execute the __complete command and only return completions
export def get_cobra_completions [
  cmd: string,  # The cobra command (e.g. kubectl)
  spans: list,  # The list of spans
] {
  # return just the completions from the result. 
  (exec_cobra_complete $cmd $spans).completions
}

let external_completer = {|spans|
  {
    $spans.0: { }
    kubectl: { get_cobra_completions 'kubectl' $spans }
    minikube: { get_cobra_completions 'minikube' $spans }
  } | get $spans.0 | each {|it| do $it}
}

As @rsteube mentioned, registration will be a little bit odd. I think that can be documented in commends for the completion generation though. It will require adding the that code block to the nushell config file ($nu.config-path) and updating config.external_completer entry:

let-env config = {
    external_completer: $external_completer
}

One nice thing is that for other cobra commands, only the external_completer block will need to be updated.

@marckhouzam, if this approach looks good, I'll update nushell_completer.go with it.

[marckhouzam]

Thanks @ayax79.
I was able to test the above script and things look promising.
However, we need to come up with a solution to enable completion for more than one tool using Cobra. Here is where I think it needs some thought.

I'm assuming that each tool, myprogram for example, will generate the above script ending with

let external_completer = {|spans|
  {
    $spans.0: { }
    myprogram: { get_cobra_completions 'myprogram' $spans }
  } | get $spans.0 | each {|it| do $it}
}

The problem is that if I have multiple tools using this, only the last sourced script will work as each one sets the same external_completer variable. What we do in Cobra is prefix each variable and function in the script by <programName>_. In this case the script would set variables such as myprogram_external_completer.

After that, we need to figure out a way to have the one

let-env config = {
    external_completer: $external_completer
}

manage to include all tools that have cobra completions.

[ayax79]

If we took this approach, it would still be possible to wrap the cobra completer and chain completers together -- it would be up to the user to figure how they wanted to handle it.

The configuration itself isn't changeable once the shell is loaded. Nushell works much more like a compiled language. Dynamic loading of resources and execution isn't really possible. The two phase execution approach is part of the reason why nushell is so fast, but it does create some awkwardness. You can see how I got around this, by creating a subshell to execute the command I assembled:

 # Since nushell doesn't have anything like eval, execute in a subshell
 let result = (do -i { nu -c $"'($full_cmd)'" } | complete)

What I really don't like about this approach is that nushell's environment has to load to execute the command (I think, need to verify). It might be feasible to do something kind of like execute completions from a directory, but it could get expensive:

let external_completer = {|spans|
    let completion_command = $"($env.COMPLETION_DIR)/($spans.0).nu ($spans | str join ' ')"
    nu -c $completion_file 
}

Using something like carapace-bin is the best use of external_completers, because carpace itself is plugable.

[marckhouzam]

Not knowing anything about nushell, I'm having trouble following the subtleties.
If you could describe the steps the user would take to get the completions to work for multiple tool, we could see if it is something that we can move forward with.

If we can reach a solution where any one tool can generate some nushell completion script and the user could add something to their configuration file to setup any number of such scripts, we would have something sufficient.

[ayax79]

I simplified the nushell external configurator to this:

# An external configurator that works with any cobra based
# command line application (e.g. kubectl, minikube)
let cobra_configurator = {|spans| 
  let cmd = $spans.0

  # skip the first entry in the span (the command) and join the rest of the span to create __complete args
  let cmd_args = ($spans | skip 1 | str join ' ') 

  # If the last span entry was empty add "" to the end of the command args
  let cmd_args = if ($spans | last | str trim | is-empty) {
    $'($cmd_args) ""'
  } else {
    $cmd_args
  }

  # The full command to be executed
  let full_cmd = $'($cmd) __complete ($cmd_args)'

  # Since nushell doesn't have anything like eval, execute in a subshell
  let result = (do -i { nu -c $"'($full_cmd)'" } | complete)

  # Create a record with all completion related info. 
  # directive and directive_str are for posterity
  let stdout_lines = ($result.stdout | lines)
  let result = ({
    completions: ($stdout_lines | drop | lines | parse "{value}\t{description}")
    directive_str: ($result.stderr)
    directive: ($stdout_lines | last)
  })

  $result.completions
}

The process would be:

1. Run <cmd> completion nushell to generate the above block
2. Add the above block to the $nu.config-path (location of the nushell config file)
3. Configure the entry external_configurator under config to be $cobra_configurator

There will be no need to change the configuration for any subsequent cobra based app. The above configuration will work with any cobra based app without any further change.

[marckhouzam]

Wow! After sourcing the script in the latest comment and then doing

let-env config = {external_completer: $cobra_configurator}

I was able to do completion for helm and kubectl!

One question with this approach: are we going to break completion for any non-cobra program that uses the external_completer?

@ayax79 Could you update the PR with the new script and then I can start testing more thoroughly?
Also, there are the shell completion directives we'll have to try to support.

One thing to investigate is that flag value completion does not seem to work:

$ kubectl --namespace [tab]
NO RECORDS FOUND

We should be getting a list of namespaces.

But this is a great start!

[ayax79]

The external completer is a fallback. Completions using externs will load first. If someone wants to use more than one external completer they could write a completer that wraps two:

let chaining_completer = {|spans|
  let completions = (do $cobra_completer $spans)
  let completions = if ($completions | is-empty) {
    (do $other_completer $spans)
  } else {
    $completions
  }
  $completions
}

I'll poke around with the flag completions. It should work the same as anything else.

[ayax79]

There was a bug in the line parsing causing it to drop entries that weren't exactly {value}\t{description}. Anything without a description was getting dropped. I changed to parse via regex. The only thing that might get dropped with this one is values (commands/flags/options) that have particularly exotic characters, I am matching [\w-.:+]. This is the corrected version:

# An external configurator that works with any cobra based
# command line application (e.g. kubectl, minikube)
let cobra_configurator = {|spans| 
 
  let cmd = $spans.0

  # skip the first entry in the span (the command) and join the rest of the span to create __complete args
  let cmd_args = ($spans | skip 1 | str join ' ') 

  # If the last span entry was empty add "" to the end of the command args
  let cmd_args = if ($spans | last | str trim | is-empty) {
    $'($cmd_args) ""'
  } else {
    $cmd_args
  }

  # The full command to be executed
  let full_cmd = $'($cmd) __complete ($cmd_args)'

  # Since nushell doesn't have anything like eval, execute in a subshell
  let result = (do -i { nu -c $"'($full_cmd)'" } | complete)

  # Create a record with all completion related info. 
  # directive and directive_str are for posterity
  let stdout_lines = ($result.stdout | lines)

  let $completions = ($stdout_lines | drop | parse -r '([\w\-\.:\+]*)\t?(.*)' | rename value description)

  let result = ({
    completions: $completions
    directive_str: ($result.stderr)
    directive: ($stdout_lines | last)
  })

  $result.completions
}

I will have an updated pull request shortly.

[marckhouzam]

This is looking great. Here are some things I noticed:

1. Could you add nushell to the default completion command? For example: https://github.com/spf13/cobra/blob/main/completions.go#L782-L785
2. The = form of flags does not quite work. When doing kubectl -n=<tab>, the -n= part is removed
3. Probably linked to the previous point, doing kubectl --namespace=<tab> then selecting a completion choice, and then pressing backpace actually crashes the shell
4. When a completion is accepted a space should be added after it. For example kubectl comple<tab> should become kubectl completion (with a space after completion) ( except if the ShellCompDirectiveNoSpace is specified)
5. Would we be able to support the ShellComp directives? Or maybe a subset of them?

   cobra/completions.go

   Lines 54 to 78 in 7bb1440

   	const (
   	// ShellCompDirectiveError indicates an error occurred and completions should be ignored.
   	ShellCompDirectiveError ShellCompDirective = 1 << iota
   	// ShellCompDirectiveNoSpace indicates that the shell should not add a space
   	// after the completion even if there is a single completion provided.
   	ShellCompDirectiveNoSpace
   	// ShellCompDirectiveNoFileComp indicates that the shell should not provide
   	// file completion even when no completion is provided.
   	ShellCompDirectiveNoFileComp
   	// ShellCompDirectiveFilterFileExt indicates that the provided completions
   	// should be used as file extension filters.
   	// For flags, using Command.MarkFlagFilename() and Command.MarkPersistentFlagFilename()
   	// is a shortcut to using this directive explicitly. The BashCompFilenameExt
   	// annotation can also be used to obtain the same behavior for flags.
   	ShellCompDirectiveFilterFileExt
   	// ShellCompDirectiveFilterDirs indicates that only directory names should
   	// be provided in file completion. To request directory names within another
   	// directory, the returned completions should specify the directory within
   	// which to search. The BashCompSubdirsInDir annotation can be used to
   	// obtain the same behavior but only for flags.
   	ShellCompDirectiveFilterDirs

6. File completion does not seem to work: if there are no completions choices, the shell should suggests file names (except if the ShellCompDirectiveNoFileComp is specified)

[github-actions]

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

[github-actions]

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

[ayax79]

I just found a nasty bug.

When running things vim, it locks up because vim interprets vim __complete as a valid command and never returns.

[ayax79]

I think I am going to have to put a whitelist in for commands to execute the completer for:

# A list of cobra that completion will be attempted for.
# Add new apps to this list to enable completion for them.
let cobra_apps = ["kubectl", "minikube"]

# An external configurator that works with any cobra based
# command line application (e.g. kubectl, minikube)
let cobra_completer = {|spans| 
  let cmd = $spans.0

  if not ($cobra_apps | where $it == $cmd | is-empty) {
    let ShellCompDirectiveError = 1
    let ShellCompDirectiveNoSpace = 2
    let ShellCompDirectiveNoFileComp = 4
    let ShellCompDirectiveFilterFileExt = 8
    let ShellCompDirectiveFilterDirs = 16
   
    let last_span = ($spans | last | str trim)

    # skip the first entry in the span (the command) and join the rest of the span to create __complete args
    let cmd_args = ($spans | skip 1 | str join ' ') 

    # If the last span entry was empty add "" to the end of the command args
    let cmd_args = if ($last_span | is-empty) {
      $'($cmd_args) ""'
    } else {
      $cmd_args
    }

    # The full command to be executed with active help disable (Nushell does not support active help)
    let full_cmd = $'($cmd)_ACTIVE_HELP=0 ($cmd) __complete ($cmd_args)'

    # Since nushell doesn't have anything like eval, execute in a subshell
    let result = (do -i { nu -c $"'($full_cmd)'" } | complete)

    # Create a record with all completion related info. 
    # directive and directive_str are for posterity
    let stdout_lines = ($result.stdout | lines)
    let directive = ($stdout_lines | last | str trim | str replace ":" "" | into int)
    let completions = ($stdout_lines | drop | parse -r '([\w\-\.:\+\=]*)\t?(.*)' | rename value description)

    # Add space at the end of each completion
    let completions = if $directive != $ShellCompDirectiveNoSpace {
      ($completions | each {|it| {value: $"($it.value) ", description: $it.description}})
    } else {
      $completions
    }

    if $last_span =~ '=$' {
      # return flag as part of the completion so that it doesn't get replaced
      $completions | each {|it| $"($last_span)($it.value)" }
    } else if $directive == $ShellCompDirectiveNoFileComp {
      # Allow empty results as this will stop file completion
      $completions
    } else if ($completions | is-empty)  or  $directive == $ShellCompDirectiveError {
      # Not returning null causes file completions to break
      # Return null if there are no completions or ShellCompDirectiveError 
      null
    } else {
      $completions
    }
  } else {
    null
  }
}

I don't think it will be too onerous. The workflow will be:

If you haven't setup cobra_completer

1. edit config.nu
2. add generated contents
3. set external_completer = cobra_completer
4. start new shell

If you have setup cobra_completer

1. Update the cobra_apps array to contain your app, for example:

let cobra_apps = ["kubectl", "minikube", "helm"]

2. restart shell

[marckhouzam]

I think I am going to have to put a whitelist in for commands to execute the completer

For other shells we don't use a "global" completer but instead have to register each program individually. So if kubectl generates the completion script, it also register kubectl (and only kubectl) to use the functions in that script.

If nushell completions don't work like that then your allow-list idea sounds like the right approach. I wonder if we could automate adding a program name to the list in the script itself?

[marckhouzam]

@ayax79 I've been thinking about how users will setup nushell completions. The shared script between programs is an efficient approach and I'll be honest that I've been tempted to do the same for the other shells. The reason I haven't is because it may cause problems in the future.

If we choose to evolve the script later on, how will we handle programs that use the new version of the script and programs that have not moved to it? For example, say that in Cobra 1.8 we decide to add a new __completeNewFeature command to cobra programs to support some new feature, and that each shell script is modified to use it. Then a program that uses cobra 1.8 will need the new script, but a program that is still on older cobra will need the old script. With a common script in nushell, this will cause problems.

I was thinking we could go back to using something that you started with (sorry about the wrong syntax):

let external_completer = {|spans|
  {
    $spans.0: { }
    kubectl: { $kubectl_completer }
    helm: { $helm_completer }
    minikube: { $minikube_completer }
  } | get $spans.0 | each {|it| do $it}
}

and have each program generate their own completion variable (instead of $cobra_completer we'd have $helm_completer, $kubectl_completer, etc).

We then would also no longer need an allow-list of cobra-programs.

And instead of inserting the entire many script into config nu, we would tell the user to do

helm completion nushell | save ~/.completions/helm

and then in config nu the user would add:

source ~/.completions/helm

The user would do this for each program.

Does that sound reasonable?

[ayax79]

Sure, sounds good. I'll get the changes in when I have some time.

[github-actions]

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

[ayax79]

There are some cosmetics comments in-line. And one more feature I forgot about: the other shells support doing completions from the middle of the line. For example:

kubectl --namespace <TAB> get pod

I think there must be a bug in nushell itself when trying to handle completions in the middle of the command. The completer is getting some really crazy span information:

When I hit right in front of pods, I get span of apod.

When I type in --name and hit , I receive a span of --namea

When I hit in whitespace between kubectl and pods I get a span of "a" and it tries to complete it.

@rsteube, have you seen anything like this before?

[marckhouzam]

I think there must be a bug in nushell itself when trying to handle completions in the middle of the command

Ok. Until that is fixed it just won't work for Cobra. It's not a blocker to get this merged.

[marckhouzam]

It is valid for a program to request file completion for a flag value. For example:

kubectl apply -f <tab>

This currently works.
However, if we use the = for it does not:

kubectl apply -f=<tab>

I'm not sure if you will be to handle that case or not. If not, we'll just have to live with it. It's not critical.

[marckhouzam]

Might as well check for ShellCompDirectiveFilterDirs here also, since it is also not supported.

[rsteube]

When I hit in whitespace between kubectl and pods I get a span of "a" and it tries to complete it.

@rsteube, have you seen anything like this before?

Yeah, got the same behaviour. Seems to be caused by nushell. So best to open an issue there.

gh <TAB> issue
# 2022/12/30 10:45:45 nushell ["carapace","_carapace","nushell","gh","a"]

If i am not mistaken this is related to space suffix determination. I think it was nushell that was adding a character (a) suffix before doing word splitting to avoid the empty word to be skipped. It is removed later but this seems to not work when invoked between words (thus the single a parameter). Then there's even a subsequent aa for me due to it's auto-invocation of completion.

see https://github.com/nushell/nushell/blob/be311829690c511edd16149b3183b8e0fef0d336/crates/nu-cli/src/completions/completer.rs#L119 and the block at line 141

[marckhouzam]

@ayax79 Just checking about this PR. I believe the ball was in your court, or am I mistaken?

[ayax79]

Hi @marckhouzam,

I know.. I have been pretty slammed with starting a new job. I am hoping to get back to it soon.

[marckhouzam]

Hi @marckhouzam,

I know.. I have been pretty slammed with starting a new job. I am hoping to get back to it soon.

No worries, I am in the same situation. I was just asking to be sure.

[Bios-Marcel]

Hey, may I ask for a summary of what is actually still to do here? Is it just what is mentioned here? #1857 (comment)

Maybe I could chime in and help. It is sad to see this sitting around here for so long :o

[marckhouzam]

Hey, may I ask for a summary of what is actually still to do here? Is it just what is mentioned here? #1857 (comment)

Maybe I could chime in and help. It is sad to see this sitting around here for so long :o

Yes I believe that’s the main part that remains.
I was even tempted to do it on my own, but I am too uncomfortable with nushell syntax. Anyone that wants to pick this up is more than welcomed

[Bios-Marcel]

Well, I don't know nushell very well either, but it is a nice cross platform shell and I see more and more people using it. So yeah, I'd have to read the docs first as well :D

Thanks for the quick reply

[ayax79]

I can try to pick it up again. It's been on the back of my mind lately.

[ayax79]

I'm also on the nushell core team now.

[github-actions]

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

[ayax79]

@marckhouzam, I have been thinking more about the direction this PR has currently taken of generating a script. A lot has matured around external completers and nushell since I first began this. Other external completers such as nushell, fish, etc handle most of the logic within the command themselves (from external completers nushell cookbook chapter):

let carapace_completer = {|spans|
    carapace $spans.0 nushell ...$spans | from json
}

let fish_completer = {|spans|
    fish --command $'complete "--do-complete=($spans | str join " ")"'
    | $"value(char tab)description(char newline)" + $in
    | from tsv --flexible --no-infer
}

What if instead of generating a rather long nushell script, if <cobra command> completions nushell actually takes in nushell span arguments, handle most of the bulk of the logic internally, and returns something easily parseable by a much smaller completer hook?

let minikube_completer = {|spans| 
   minikube completions nushell $spans | from json
}

thoughts?