Tab-completion for remote paths #1529

adamboche · 2017-07-14T22:36:49Z

I believe rsync and scp can tab-complete remote paths. It would be nice to have this in rclone too. Here's some Stack Exchange discussion of the issue.

The text was updated successfully, but these errors were encountered:

ncw · 2017-07-15T03:26:25Z

What this needs is is magic in the auto completion script. This would have to call rclone to get a listing of the remote at the correct time.

I could really do with help on this one as bash auto completion is a bit of a mystery to me!

diamondsw · 2017-07-17T15:25:55Z

If you manage to get this working (it would be quite magical!), please look at crypt'ed remotes, as these are particularly difficult to navigate at present by any means other than a fuse mount.

Skylion007 · 2017-07-17T21:36:12Z

I have a little experience bash scripting. The main issue with this is how portable rclone is. To get proper bash completion support you need to have a script which loads the autocompletion function into the bash environment. Normally this is done by putting the script in a specific folder. You could also include a bash script with go or perhaps the best solution is to have rclone modify the necessary environment variables with a flag (example: rclone --init_autocomplete) for example. There are also clever approaches to abuse go packages to manage autocomplete for you.

The second approach is probably the best one to use. rclone would need to be able to "install" and uninstall it's autocompletion scripts though, that could be easily done with command flags though.

danemacmillan · 2018-02-05T20:16:29Z

I'd be happy to get tab completion for just the names of remotes to get things started. For example, when typing rclone mount [tab], it will suggest all the remotes from my .rclone.conf file.

aurelg · 2018-09-08T05:50:39Z

Could this help? (Reddit discussion here)

ncw · 2018-09-11T09:46:39Z

@aurelg that looks very interesting.

@cspeterson would you like to contribute your rclone tab completion script to rclone?

Rclone has a bash completion script already generated by cobra: https://rclone.org/commands/rclone_genautocomplete_bash/ so we'd have to figure out how to join the two together.

cspeterson · 2018-09-11T13:55:48Z

@ncw I'd be happy to work on that. Question though: do all possible rclone targets construct paths the same way? I only designed and tested my script to work with GDrive and it makes some assumptions there

danemacmillan · 2018-09-11T15:16:24Z

@cspeterson A couple things I noted:

grep -P is only available in GNU, so by default, this will not work on MacOS/BSD/Unix. It's a trivial fix via brew install grep and adding the GNU version to PATH; oddly, it's not part of coreutils.
Pass the --quiet or -q options, so that tabbing does not output verbose rclone information, especially for users who set environment variables like export RCLONE_VERBOSE=1.

Also, I'm super-excited for this to get into the core.

cspeterson · 2018-09-11T15:25:47Z

@danemacmillan

grep: I overlooked that grep, I thought I had everything in pure bash! I can either reimplement that call in bash or modify to work without the GNU flag, sure.
quiet: Good call 👍

ncw · 2018-09-11T16:54:23Z

@cspeterson wrote:

I'd be happy to work on that

:-)

Question though: do all possible rclone targets construct paths the same way? I only designed and tested my script to work with GDrive and it makes some assumptions there

Yes all remotes construct paths in the same way remote:path/path/path. A small number of remotes will do something different if you do remote:/path/path/path (eg sftp which will look from the root rather than from the users home directory).

olalonde · 2018-10-14T16:50:45Z

Bump :) I believe there are similar auto completion scripts for scp

sgtpep · 2019-01-29T11:03:51Z

Here's my version of completion for remotes (heavily based on @cspeterson's script): https://raw.githubusercontent.com/sgtpep/dotfiles/master/.bash_completion. Should play fine with rclone's own completion and doesn't add trailing space after every completion. Tested only on Linux with fairly recent version of bash.

ncw · 2019-01-29T14:40:19Z

Here's my version of completion for remotes (heavily based on @cspeterson's script): https://raw.githubusercontent.com/sgtpep/dotfiles/master/.bash_completion. Should play fine with rclone's own completion and doesn't add trailing space after every completion. Tested only on Linux with fairly recent version of bash.

Nice one! How do you install it? Do you want to contribute this to the rclone repo?

cspeterson · 2019-01-29T14:49:12Z

@sgtpep nice! I'll be using your update for sure

this reminder brings me to share that I spent some time struggling with getting this script into the project but i hit a wall at some point and haven't found the time since. Perhaps @sgtpep is better equipped to get this done in a timely fashion? ¯\(ツ)/¯

sgtpep · 2019-01-29T16:21:51Z

@ncw, @cspeterson Thanks!

How do you install it?

For now I just keep https://raw.githubusercontent.com/sgtpep/dotfiles/master/.bash_completion as ~/.bash_completion and source it from ~/.bashrc like this: . ~/.bash_completion

Do you want to contribute this to the rclone repo?

I'd love to. It probably requires some modifications to make it compatible with bash 3.* from macOS and having a fresh look at some other possible edge cases. I hope to have time for this in a week or so. If anyone wants to do it sooner, that would be great.

ncw · 2019-01-29T17:40:07Z

I'm not quite sure of the best way of incorporating this into rclone... It could be as part of the rclone genautocomplete command, but I'd need some guidance as to where to put it!

sgtpep · 2019-02-06T01:29:11Z

@ncw Sorry for the delay. This seems to work on my /usr/share/bash-completion/completions/rclone: https://gist.github.com/sgtpep/c95a50f8980a143bef8f7176d9013710. It supposedly should work on macOS/Homebrew, but couldn't test.

ncw · 2019-02-06T11:29:53Z

I managed to work out (eventually!) how to add this to the cobra framework bash completion generator. I

Can you give it a go here? rclone genautocomplete bash to generate the script. (If running with sudo make sure that you are running the correct rclone binary, don't do what i just did ;-)

https://beta.rclone.org/branch/v1.45-175-g871ae1f1-fix-1529-bash-completion-beta/ (uploaded in 15-30 mins)

Here is the commit 871ae1f to show you where it is. I realise I've forgotten to attribute @sgtpep so I'll change that after a round of feedback.

Do you think this should be on by default? It seems to work very well, but it does have the downside that it interferes with local completions.

What other testing does it need?

sgtpep · 2019-02-06T14:31:14Z

but it does have the downside that it interferes with local completions.

Indeed, updated gist. Here is the diff of that diff :) https://gist.github.com/sgtpep/c95a50f8980a143bef8f7176d9013710/revisions.

What other testing does it need?

Has anyone tried it on stock macOS bash 3.* and/or recent bash from Homebrew?

ncw · 2019-02-06T21:38:36Z

but it does have the downside that it interferes with local completions.

Indeed, updated gist. Here is the diff of that diff :) https://gist.github.com/sgtpep/c95a50f8980a143bef8f7176d9013710/revisions.

That is great - thank you! Merging the completions for the remotes and the local file names is brilliant!

I wish I understood how it worked - it all just looks like magic to me ;-)

I integrated the changes into rclone here

https://beta.rclone.org/branch/v1.45-177-ga969e259-fix-1529-bash-completion-beta/ (uploaded in 15-30 mins)

What other testing does it need?

Has anyone tried it on stock macOS bash 3.* and/or recent bash from Homebrew?

I tried to find a linux server with bash3 but looks like the distros went to bash4 a long time ago!

Unfortunately I don't have a mac.

Does anyone else want to give it a test?

@sgtpep Do you think I should merge this for the v1.46 release (which is due at the weekend)?

sgtpep · 2019-02-06T22:12:07Z

Thank you! It looks all good to me. I'd really love to see this change in the new release, and my guess is that that addition won't break or affect the rest of completion code, at least other locations of completion should remain working.

The problem with macOS is not only really vintage bash which some people replace with a version from Homebrew. Homebrew provides (or provided) a version of bash-completion that is different from one on linuxes and lacks _init_completion. That's why there's a call to __rclone_init_completion (saw it somewhere in rclone's completion file).

ncw · 2019-02-07T10:13:54Z

Thank you! It looks all good to me. I'd really love to see this change in the new release, and my guess is that that addition won't break or affect the rest of completion code, at least other locations of completion should remain working.

Great!

The problem with macOS is not only really vintage bash which some people replace with a version from Homebrew. Homebrew provides (or provided) a version of bash-completion that is different from one on linuxes and lacks _init_completion. That's why there's a call to __rclone_init_completion (saw it somewhere in rclone's completion file).

I put a call for testers on the forum.

ncw · 2019-02-08T19:02:36Z

I've merged this to master in 60a4a8a now which means it will be in the latest beta in 15-30 mins and released in v1.46

Thank you all for your help, especially @cspeterson and @sgtpep

@cspeterson

Thanks to: - Christopher Peterson (@cspeterson) for the original script - Danil Semelenov (@sgtpep) for many refinements

felder · 2019-02-12T00:54:57Z

This seems to conflict with bash completion for kubectl on homebrew for osx.

With rclone bash completion active, kubectl get tab tab no longer works.

ncw · 2019-02-12T12:30:21Z

This seems to conflict with bash completion for kubectl on homebrew for osx.

With rclone bash completion active, kubectl get tab tab no longer works.

Do the rclone completions work on their own?

Which version of bash are you using? bash --version

@sgtpep any ideas?

sgtpep · 2019-02-12T14:08:39Z

@ncw I was able to reproduce it downloading recent binaries of rclone and kubectl and generating ~/.bash_completion combining both completions:

~/Downloads/rclone genautocomplete bash ~/.bash_completion
~/Downloads/kubectl completion bash >> ~/.bash_completion

And kubectl getTab gets completions for rclone's remotes.

Looks like there is a name clash between function from both completion scripts. They both contain functions named __custom_func, and the latest declaration takes precedence (rclone's, alphabetically). I'm not sure if this name has special meaning in bash-completion, but ideally, of course, both projects should've namespaced them with their prefixes. I'm sorry if it's me who overlooked this!

ncw · 2019-02-12T22:02:49Z

@ncw I was able to reproduce it downloading recent binaries of rclone and kubectl and generating ~/.bash_completion combining both completions:
~/Downloads/rclone genautocomplete bash ~/.bash_completion
~/Downloads/kubectl completion bash >> ~/.bash_completion
And kubectl getTab gets completions for rclone's remotes.

Looks like there is a name clash between function from both completion scripts. They both contain functions named __custom_func, and the latest declaration takes precedence (rclone's, alphabetically). I'm not sure if this name has special meaning in bash-completion, but ideally, of course, both projects should've namespaced them with their prefixes. I'm sorry if it's me who overlooked this!

Ah well spotted.

It looks like this is was an existing cobra issue: spf13/cobra#694 which has been fixed by spf13/cobra#730 However because of spf13/cobra#259 cobra hasn't had a release so rclone is still using the old version of cobra without that support. (I did try the __rclone_custom_func but it never got called).

So this should be quite easy to fix with a new cobra version and a minor tweak

diff --git a/cmd/help.go b/cmd/help.go
index 59adc489a..066963fff 100644
--- a/cmd/help.go
+++ b/cmd/help.go
@@ -37,7 +37,7 @@ documentation, changelog and configuration walkthroughs.
 
 const (
 	bashCompletionFunc = `
-__custom_func() {
+__rclone_custom_func() {
     if [[ ${#COMPREPLY[@]} -eq 0 ]]; then
         local cur cword prev words
         if declare -F _init_completion > /dev/null; then

Before this change, rclone used the `__custom_func` hook to control the completions of remote files. However this clashes with other cobra users, the most notable example being kubectl. Upgrading cobra to master allows us to use a namespaced function `__rclone_custom_func` which fixes the problem. Fixes #1529

ncw · 2019-02-13T14:35:18Z

@felder I've put a fix for this here

https://beta.rclone.org/branch/v1.46.0-019-g8209f321-fix-1529-completions-beta/ (uploaded in 15-30 mins)

Run sudo rclone genautocomplete bash and re-open your terminal to test.

felder · 2019-02-13T18:21:29Z

Looks good.

kubectl get tab tab does the right thing now as does rclone tab tab

Thanks for the quick fix!

Before this change, rclone used the `__custom_func` hook to control the completions of remote files. However this clashes with other cobra users, the most notable example being kubectl. Upgrading cobra to master allows us to use a namespaced function `__rclone_custom_func` which fixes the problem. Fixes #1529

ncw · 2019-02-13T23:03:32Z

Looks good.

kubectl get tab tab does the right thing now as does rclone tab tab

Thanks for the quick fix!

Thanks for testing.

I've merged this to master now which means it will be in the latest beta in 15-30 mins and released in v1.47

aureliendavid · 2019-03-13T13:55:49Z

Hi,

The bash completion seems to work but doesn't escape remote path containing spaces or special characters (like parenthesis).

$ rclone lsd remote:/test/path<tab>

completes to

$ rclone lsd remote:/test/path with spaces  (instead of /test/path\ with\ spaces)

Also it doesn't complete if the remote path is quoted although rclone does seem to support quoted remote paths:

$ rclone lsd remote:"/test/path<tab> # doesn't complete here

(should I open an issue for this maybe?)

thanks

ncw · 2019-03-13T15:15:09Z

@aureliendavid Can you make a new issue about this please :-)

@cspeterson

Thanks to: - Christopher Peterson (@cspeterson) for the original script - Danil Semelenov (@sgtpep) for many refinements

Before this change, rclone used the `__custom_func` hook to control the completions of remote files. However this clashes with other cobra users, the most notable example being kubectl. Upgrading cobra to master allows us to use a namespaced function `__rclone_custom_func` which fixes the problem. Fixes rclone#1529

@cspeterson

Thanks to: - Christopher Peterson (@cspeterson) for the original script - Danil Semelenov (@sgtpep) for many refinements

Before this change, rclone used the `__custom_func` hook to control the completions of remote files. However this clashes with other cobra users, the most notable example being kubectl. Upgrading cobra to master allows us to use a namespaced function `__rclone_custom_func` which fixes the problem. Fixes rclone#1529

NightMachinery · 2020-07-10T07:57:35Z

It seems this feature is not supported for zsh autocompletion?

NightMachinery · 2020-07-10T07:59:33Z

In bash, the completions don't seem to cache. Which makes it very slow.

ncw · 2020-07-10T15:29:55Z

It seems this feature is not supported for zsh autocompletion?

In theory rclone genautocomplete zsh should work. In practice it has some shortcomings I believe (not a zsh user).

In bash, the completions don't seem to cache. Which makes it very slow.

Is there a standard way of caching completions?

NightMachinery · 2020-07-10T16:32:42Z

@ncw

In theory rclone genautocomplete zsh should work. In practice it has some shortcomings I believe (not a zsh user).

It completes the options, but not the remote paths.

Is there a standard way of caching completions?

I don't know, I am not familiar with the completion systems. I just wrote a custom script for myself that caches rclone lsf --recursive and use fzf together with the cache to copy files to my local machine.

# eval-memoi caches the command in redis
rclonef () {
	local paths=("${(@f)$(memoi_expire=${rfExpire:-$((3600*24))} eval-memoi rclone lsf --recursive "${1}:" | fzf)}")
	local i
	for i in $paths[@]
	do
		rclone copy "${1}:$i" "$2"
	done
}

ncw · 2020-07-11T07:52:44Z

I just wrote a custom script for myself that caches rclone lsf --recursive and use fzf together with the cache to copy files to my local machine.

That is a nice solution 😃It sounds a bit specialized to add to rclone though.

eoinkennedy · 2022-10-01T20:06:18Z

Two years latter and still no resolution? So many partial solutions, but nothing built-in. Right now, after typing rclone copy and then "a remote": (I celebrate to remember), then TAB twice, and the listing I get is from the directory that I am in at local storage.

ncw · 2022-10-03T10:31:40Z

@eoinkennedy this works fine for me with rclone genautocomplete bash

$ rclone lsf drive:test/200 <TAB><TAB>
$ rclone lsf drive:test/2001

$ rclone lsf drive:test/2001 <TAB><TAB>
2001-05-28-115008.jpg  2001test3/             
2001test/              2001test5/             
$ rclone lsf drive:test/2001

mendy-lp · 2023-02-15T18:07:19Z

@ncw
I was not able to get tab completion for remote paths on zsh.
Command completion and remote name completion work just fine. The only thing that is not working is tab-completion for remote paths,

Btw I tested it (remote path completion) on bash and it works perfectly, so the issue seems to be with the zsh completion

ncw added the enhancement label Jul 15, 2017

ncw added this to the Help Wanted milestone Feb 10, 2018

ncw added the help wanted label Feb 10, 2018

ncw modified the milestones: Help Wanted, v1.46 Feb 8, 2019

ncw closed this as completed Feb 8, 2019

ncw added a commit that referenced this issue Feb 8, 2019

genautocomplete: add remote path completion for bash - fixes #1529

60a4a8a

Thanks to: - Christopher Peterson (@cspeterson) for the original script - Danil Semelenov (@sgtpep) for many refinements

gcornut pushed a commit to gcornut/rclone that referenced this issue Mar 15, 2019

genautocomplete: add remote path completion for bash - fixes rclone#1529

14de8c0

Thanks to: - Christopher Peterson (@cspeterson) for the original script - Danil Semelenov (@sgtpep) for many refinements

gcornut pushed a commit to gcornut/rclone that referenced this issue Mar 15, 2019

genautocomplete: add remote path completion for bash - fixes rclone#1529

e16cea4

Thanks to: - Christopher Peterson (@cspeterson) for the original script - Danil Semelenov (@sgtpep) for many refinements

Tab-completion for remote paths #1529

Tab-completion for remote paths #1529

Comments

adamboche commented Jul 14, 2017

ncw commented Jul 15, 2017

diamondsw commented Jul 17, 2017

Skylion007 commented Jul 17, 2017 • edited

danemacmillan commented Feb 5, 2018

aurelg commented Sep 8, 2018 • edited

ncw commented Sep 11, 2018

cspeterson commented Sep 11, 2018 • edited

danemacmillan commented Sep 11, 2018 • edited

cspeterson commented Sep 11, 2018

ncw commented Sep 11, 2018

olalonde commented Oct 14, 2018 • edited

sgtpep commented Jan 29, 2019

ncw commented Jan 29, 2019

cspeterson commented Jan 29, 2019 • edited

sgtpep commented Jan 29, 2019

ncw commented Jan 29, 2019

sgtpep commented Feb 6, 2019

ncw commented Feb 6, 2019

sgtpep commented Feb 6, 2019 • edited

ncw commented Feb 6, 2019

sgtpep commented Feb 6, 2019

ncw commented Feb 7, 2019

ncw commented Feb 8, 2019 • edited

felder commented Feb 12, 2019 • edited

ncw commented Feb 12, 2019

sgtpep commented Feb 12, 2019 • edited

ncw commented Feb 12, 2019

ncw commented Feb 13, 2019

felder commented Feb 13, 2019

ncw commented Feb 13, 2019

aureliendavid commented Mar 13, 2019

ncw commented Mar 13, 2019

NightMachinery commented Jul 10, 2020

NightMachinery commented Jul 10, 2020

ncw commented Jul 10, 2020

NightMachinery commented Jul 10, 2020

ncw commented Jul 11, 2020

eoinkennedy commented Oct 1, 2022

ncw commented Oct 3, 2022

mendy-lp commented Feb 15, 2023 • edited

Skylion007 commented Jul 17, 2017 •

edited

aurelg commented Sep 8, 2018 •

edited

cspeterson commented Sep 11, 2018 •

edited

danemacmillan commented Sep 11, 2018 •

edited

olalonde commented Oct 14, 2018 •

edited

cspeterson commented Jan 29, 2019 •

edited

sgtpep commented Feb 6, 2019 •

edited

ncw commented Feb 8, 2019 •

edited

felder commented Feb 12, 2019 •

edited

sgtpep commented Feb 12, 2019 •

edited

mendy-lp commented Feb 15, 2023 •

edited