Let the commands store flagComp functions internally (and avoid global state)

[maxlandon]

Problem

Currently, flag completion functions are a global map, preventing garbage collection
to work properly, especially if the commands using these funcs are garbage-collected themselves.
This might happen if commands are declared, bound and used through yielder functions.

This is also produce more coherent API usage in different cases:

• Declare flag completions close to where you declare the flags themselves,
• When a code path happens to have access to a given command, it knows how to bind flags to it (in a more dynamic manner, for that respect).

In addition, and while in theory nothing prevents a given command cmd1 to be called with
RegisterFlagsCompletion(cmd2Flag), and still store and work correctly with another command
flag.

Changes

Add the old flagCompletionFunctions map and its mutex as unexported fields of the Command.

[CLAassistant]

All committers have signed the CLA.

[marckhouzam]

Thanks @maxlandon for trying to improve things. However, this is actually how the original implementation tried to do it in #1423 which broke flag registration as described in #1437 and fixed in #1438

[maxlandon]

Thanks for pointing those to me.
After examination and comparing with my approach, it seems that there is a very small - but very important - difference between the failed attempts. See following/preceding code comments.
Sincerely,

[maxlandon]

@marckhouzam thanks for the quick reply in any case, I see that you guys are very busy on this repo.

I am also sorry for unearthing this thing, and doubly sorry to insist on the need for this to be solved out a way or another.
This PR has been opened in relation to a larger aim, which is to be able to use the carapace engine as a library, and the latter project is on track to provide exceptional completion possibilities and cross-shell portability.

I will open a thread dedicated to this, to expose more things in detail about that.

In any case, I would be very, very, very grateful if it was possible to gather a little bit of attention once again on the present issue.

Sincerely,

[Luap99]

This approach work for most cases but it will not work for persistent flags right now.
If you have a persistent flag defined on the root cmd you do not get that completion on a child command. In order to support that you would need to walk the tree up to root command when you lookup the completion function.

[maxlandon]

True, I completely forgot this initialization problem.

I will look at it this weekend, but normally I just need to add initialization in the related functions.

[maxlandon]

Normally the initialization problem should be fixed with the last commit I pushed.

Not sure how to solve for persistent root command' flags, but working on this.

[maxlandon]

Hello @marckhouzam, sorry to ping but I think this is very urgent:
I forgot to warn when you merged #1943 that we are still actually trying to avoid global state, and the corresponding PR did not take this into account.

So in this PR and in the last 3-4 commits, the following changes/fixes have been made:

• Flag comp registration & getters are accessible through any command as a method, and when not found on it, its parents (if any) are looked up recursively. The function signatures don't change at all, they are just methods now.
• Ensure flag completion maps are always initialized before use.
• Merged with cobra master latest.

You would be very, very kind to review and/or test this as quick as possible, so that we can avoid too many people using the global flag comp getters in their code.
Thanks a lot !

Related to #2019
Also pinging @Luap99 , since you seem to be very in touch with what should and what should not be done. It seems you are able to verify very quickly if recursive/parent-root flag completions do work in this setup or not.
Thanks for any help you'll bring on this !

Ping me ruthlessly if needed for anything, at any time.

[marckhouzam]

I missed this. Sorry.
We’re going to have to look at this before the next release

[marckhouzam]

@maxlandon This does not compile

[maxlandon]

@marckhouzam i'm going to fix this later today and will ping back

[marckhouzam]

@maxlandon I have added extra tests in #2053 to protect us against regressions.
Please make sure your changes pass on top of #2053.

[maxlandon]

@maxlandon I have added extra tests in #2053 to protect us against regressions.

Please make sure your changes pass on top of #2053.

Thanks a lot @marckhouzam , could you compile the code in this PR ? Or do you still need a fix ?

Away from my laptop today unfortunately, but can be available to help for the rest of the week and beyond.

Edit: I can take care of merging the various branches on which tests have been added, as well any other for that matter.

Edit 2: I can probably do the whole thing in a new PR, with everything gathered/fixed/documented in one place.
Branch name like "completion-state-no-global"

[marckhouzam]

Thanks a lot @marckhouzam , could you compile the code in this PR ? Or do you still need a fix ?

I got it to compile (see small diff at bottom [*]), but I get a panic running the tests.

Edit: I can take care of merging the various branches on which tests have been added, as well any other for that matter.

Just rebase your PR on top of #2053 as two commits.

Edit 2: I can probably do the whole thing in a new PR, with everything gathered/fixed/documented in one place. Branch name like "completion-state-no-global"

No need for a new PR. Please just squash all your changes into one commit and force-push.

[*] patch to fix compilation:

diff --git a/completions.go b/completions.go
index 0ee4f28..c566268 100644
--- a/completions.go
+++ b/completions.go
@@ -190,10 +190,8 @@ func (c *Command) initializeCompletionStorage() {
 		c.flagCompletionMutex = new(sync.RWMutex)
 	}

-	var completionFn func(cmd *Command, args []string, toComplete string) ([]string, ShellCompDirective)
-
 	if c.flagCompletionFunctions == nil {
-		c.flagCompletionFunctions = make(map[*flag.Flag]completionFn, 0)
+		c.flagCompletionFunctions = make(map[*flag.Flag]func(*Command, []string, string) ([]string, ShellCompDirective), 0)
 	}
 }

[marckhouzam]

@maxlandon Can you clarify (I still don't understand, sorry), why we need to avoid global state.

Currently, flag completion functions are a global map, preventing garbage collection to work properly, especially if the commands using these funcs are garbage-collected themselves. This might happen if commands are declared, bound and used through yielder functions.

Ok, that make sense, but it is not dramatic, I think.

This is also produce more coherent API usage in different cases:

• Declare flag completions close to where you declare the flags themselves,

This is already how it is done. A developer registers the flag completion right next to the flag declaration. They don't know how the registration is stored in cobra.

• When a code path happens to have access to a given command, it knows how to bind flags to it (in a more dynamic manner, for that respect).

I don't get this one. Can you give an example.

In addition, and while in theory nothing prevents a given command cmd1 to be called with RegisterFlagsCompletion(cmd2Flag), and still store and work correctly with another command flag.

I can't grok that sentence. Do you want to be able to do this, or are you trying to avoid it?
Currently, it can happen but only if cmd1 has a flag also named cmd2Flag.

[maxlandon]

@marckhouzam below is the main, summed up reason for this, followed by the 3 points aiming to illustrate /clarify your last reply.

Main reason: in short, having global state (for completions or else) is in effect making an assumption about the lifecycle of commands and their completions. This assumption is impractical, useless and risky, while not being improving clarity of code related to our present matter.

1) Garbage collection (ex: closed loop applications)
In the current state of things, it is always assumed that commands will be declared/allocated/executed only once in the lifetime of the program (traditional, single-exec CLI).
However, if you consider this console library and the way it uses yielder functions to cobra commands (for execution and completion, you will notice that having global flagcomp maps is preventing correct garbage collection.
See this section of the documentation for a quick example.
See also this issue for another example where this is a problem (and related to the other carapace proposal of mine)

2) Flags compFuncs binding close to command declaration
Yes it's true that people will bind the compFuncs close to the command declaration. So consider the following:

// Current (global state)
cmd := cobra.Command{}
cobra.RegisterFlagCompletion(myFunc)

// Without global state, directly through the command method.
cmd := cobra.Command{}
cmd.RegisterFlagCompletion(myFunc)

From above you can see that:

• Point 1) is solved
• We don't need to use the cobra package reference for binding our compFunc.
• See 3) for how this is clearer on more complex scenarii.

3) Parent/child commands and their compFuncs
The advantage of binding compFuncs directly to commands is that we get a clearer/more granular way of binding completions so specific commands. Easier to read, and no impact on the completions lookup when these are being served (because the flagComp getters will look up to the command's parent when it does not have the target compFunc).

Consider this:

// root command has a flag and its completion.
parent := cobra.Command{}
parent.RegisterFlagCompletion("myflag", flagFuncParent)

// Child command has other flags and completions
child := cobra.Command{}
child.RegisterFlagCompletion("childflag", childCompFunc)

// No let's query the compfuncs
parentComp := child.GetFlagComp("myflag") // recursively looks up to the parent until found
childComp := child.GetFlagComp("childflag") // immediately found

notFoundComp := parent.GetFlagComp("childflag") // not working since lookup is always up the tree. This is conforming with how cobra looks up the command trees when devising what to complete.

4) the sentence you can't grok
Forget about this, the three points above are meant to clarify everything. Hopefully they do...

[maxlandon]

Thanks a lot @marckhouzam , could you compile the code in this PR ? Or do you still need a fix ?

I got it to compile (see small diff at bottom [*]), but I get a panic running the tests.

Edit: I can take care of merging the various branches on which tests have been added, as well any other for that matter.

Just rebase your PR on top of #2053 as two commits.

Edit 2: I can probably do the whole thing in a new PR, with everything gathered/fixed/documented in one place. Branch name like "completion-state-no-global"

No need for a new PR. Please just squash all your changes into one commit and force-push.

[*] patch to fix compilation:

diff --git a/completions.go b/completions.go

index 0ee4f28..c566268 100644

--- a/completions.go

+++ b/completions.go

@@ -190,10 +190,8 @@ func (c *Command) initializeCompletionStorage() {

 		c.flagCompletionMutex = new(sync.RWMutex)

 	}



-	var completionFn func(cmd *Command, args []string, toComplete string) ([]string, ShellCompDirective)

-

 	if c.flagCompletionFunctions == nil {

-		c.flagCompletionFunctions = make(map[*flag.Flag]completionFn, 0)

+		c.flagCompletionFunctions = make(map[*flag.Flag]func(*Command, []string, string) ([]string, ShellCompDirective), 0)

 	}

 }

I did not see this message (i'm on my phone atm). Will do as indicated !

[maxlandon]

Hello @marckhouzam , sorry I've been very very busy these days.

Taking care of this, it will be done and ready by the end of the week at worst, you have my word.

Sorry for slowing things here...

[marckhouzam]

@marckhouzam below is the main, summed up reason for this, followed by the 3 points aiming to illustrate /clarify your last reply.

Main reason: in short, having global state (for completions or else) is in effect making an assumption about the lifecycle of commands and their completions. This assumption is impractical, useless and risky, while not being improving clarity of code related to our present matter.

1) Garbage collection (ex: closed loop applications)

That makes sense.

2) Flags compFuncs binding close to command declaration Yes it's true that people will bind the compFuncs close to the command declaration. So consider the following:

// Current (global state)
cmd := cobra.Command{}
cobra.RegisterFlagCompletion(myFunc)

// Without global state, directly through the command method.
cmd := cobra.Command{}
cmd.RegisterFlagCompletion(myFunc)

Actually, your second example is how things are done today, but the cmd.RegisterFlagCompletion(myFunc) calls the global map internally.

3) Parent/child commands and their compFuncs The advantage of binding compFuncs directly to commands is that we get a clearer/more granular way of binding completions so specific commands. Easier to read, and no impact on the completions lookup when these are being served (because the flagComp getters will look up to the command's parent when it does not have the target compFunc).

Consider this:

// root command has a flag and its completion.
parent := cobra.Command{}
parent.RegisterFlagCompletion("myflag", flagFuncParent)

// Child command has other flags and completions
child := cobra.Command{}
child.RegisterFlagCompletion("childflag", childCompFunc)

// No let's query the compfuncs
parentComp := child.GetFlagComp("myflag") // recursively looks up to the parent until found
childComp := child.GetFlagComp("childflag") // immediately found

notFoundComp := parent.GetFlagComp("childflag") // not working since lookup is always up the tree. This is conforming with how cobra looks up the command trees when devising what to complete.

I believe we can already do this using the new cmd.GetFlagCompletionByName("myflag") no?
The part we don't do is recursively look up for the flag from the parent, but this can easily be added with the current API.

[maxlandon]

Hello @marckhouzam , I was actually working on this at this moment !

So to "answer your answer":
1 - Perfect, glad to have made my point successfully.
2 - Yes indeed, the RegisterFlagCompletion is a method using the global map, and therefore we're about to change the global map from a per-command one.
3 - Quite the same as 2), and the present/to-come changes are about recursively checking commands/parents flagComp maps.

More commits to solve all of this in a few minutes/hours !

[maxlandon]

Actually, to answer your third point (GetCompletionFlagByName() methods):
You will notice that in the present state of things we have one method and one function:

Global function, querying the global map thanks to a flag pointer:

cobra/completions.go

Line 149 in 890302a

func GetFlagCompletion(flag *pflag.Flag) (func(cmd *Command, args []string, toComplete string) ([]string, ShellCompDirective), bool) {

A command method, querying the global map thanks to a flag name:

cobra/completions.go

Line 158 in 890302a

func (c *Command) GetFlagCompletionByName(flagName string) (func(cmd *Command, args []string, toComplete string) ([]string, ShellCompDirective), bool) {

So with the changes in the current PR, the global function GetFlagCompletion(flag *pflag.Flag) becomes a method of the command, so that in effect people have two different ways of querying compFuncs.
Internally, the logic is the same for both the GetFlagCompletion(pointer) and GetFlagCompletionByName(): they will recursively walk up their parents' tree until the corresponding flagCompFunc is found.

[maxlandon]

Okay normally this should be good. Summarizing the work done and steps taken below:

1. Merge the main branch into this one, to get all new completion test code.
2. Change GetFlagCompletion(flagPtr *flag.Flag) to a method of the Command type.
3. For both GetFlagCompletion(ptr) and GetFlagCompletionByName(string), recursively walk the parents until the corresponding flagCompFunc is found. Return it along with a boolean.
4. Add some internal utility functions for ensuring all private Command maps/mutexes are initialized before being used.
5. Adapt the cmd.getCompletions() function (used when the __complete command is called) to actually use the cmd.GetFlagCompletion(ptr) method, which works in way consistent with how cobra traverses/looks up the command tree when devising what to complete.
6. Comment all the new functions, internal or external-facing.
7. Pass all tests.

Will pinpoint these numbers in the commits code, if you want to have visual confirmation for all of them.

[maxlandon]

Change this to a command method.

[maxlandon]

3. Recursively walk up the command parents' tree until flagComp is found.
   Behavior identical regardless of if the flagComp is searched with a flag pointer, or a flag name.

[maxlandon]

5. Use our new recursive methods to lookup what to complete when __complete is called.
   Passes all tests

[marckhouzam]

Thanks for the update @maxlandon !
Following the issues you describe here I have posted #2063 to fix the API of #1943 (and fix a bug).
This should avoid us painting ourselves in a corner while waiting for this PR here to get merged. You may want to have a look to confirm it addresses your immediate worries about propagating a global state.

As a release might be cut in the next hours, I prefer to play it safe and start with #2063 to avoid rushing your PR (which is more complicated and requires a more detailed review, as you saw from the repeated required rework.)

[maxlandon]

Thanks for the update @maxlandon ! Following the issues you describe here I have posted #2063 to fix the API of #1943 (and fix a bug). This should avoid us painting ourselves in a corner while waiting for this PR here to get merged. You may want to have a look to confirm it addresses your immediate worries about propagating a global state.

As a release might be cut in the next hours, I prefer to play it safe and start with #2063 to avoid rushing your PR (which is more complicated and requires a more detailed review, as you saw from the repeated required rework.)

Doing the review on the other PRs at the moment.
In the meantime may I request @jpmcb review on this one as well ?
I feel this PR is not that complicated, as shown by the number of additions/deletions.

I would still pretend that it is better to review/accept/reject this one before any release.
But I might be overly pretentious here...

[maxlandon]

@marckhouzam could you just allow the workflows to run once on this one ? just to make sure I'm not defending something that doesn't pass the bar :) !
Thanks a lot

EDIT: Nevermind, just saw that you ran them 15 minutes ago !

[jpmcb]

In the meantime may I request @jpmcb review on this one as well ?

Thanks for the ping: I'd be happy to help get this through. Looking now

[jpmcb]

Overall, this looks solid. I'm also going to spend some time testing this against kubectl and it's completions to ensure we're not missing / breaking anything

[marckhouzam]

@maxlandon can you rebase please

[github-actions]

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

[github-actions]

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

[maxlandon]

@marckhouzam (and @jpmcb for that matter here)

Even better than a rebase, I renamed GetFlagCompletionByName() func to GetFlagCompletionFuncByName() for the sake of consistency, and suddenly, running the tests raised a significant logic mistake in the way this function would perform the lookup.

Now everything is solved, logic is good for both functions.
Following is the comment of this new little change.

Hopeful to see this go through ! I appreciate the attention you all have given to this thing recently ! Thanks a lot !