Too many auto import quick fixes returned

[mjbvz]

Type: Bug

1. Open a python file and type a
2. Trigger quick fixes by clicking on the lightbulb

Bug
I get tons quick fixes for importing what seems like every possible package

Extension version: 2023.1.41
VS Code version: Code - Insiders 1.75.0-insider (Universal) (270c3b12906a88e9c67fc5ddef87be6aa3b75551, 2023-01-25T05:23:34.922Z)
OS version: Darwin x64 22.2.0
Modes:
Sandboxed: Yes

System Info

Item	Value
CPUs	Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz (16 x 2400)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled metal: disabled_off multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_renderer: enabled_on video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: disabled_off
Load (avg)	3, 3, 3
Memory (System)	32.00GB (1.28GB free)
Process Argv	--crash-reporter-id 48781ca2-1705-4f64-9bab-325055aab55d
Screen Reader	no
VM	0%

[rchiodo]

Is there a set of filters you would want to apply? It's only filtering on the letter 'a' and everything that matches.

However, you can control the set of things we index with these settings here:
https://github.com/microsoft/pylance-release/blob/main/README.md#settings-and-customization under the python.analysis.packageIndexDepths section.

Something like so would eliminate everything except say django:

  "python.analysis.packageIndexDepths": [
    {
      "name": "",    // This entry turns off symbols for everything
      "depth": 0,
      "includeAllSymbols": false 
    },
    {
      "name": "django", // This adds it back for just django
      "depth": 4,
      "includeAllSymbols": true
    }
  ],

[bschnurr]

Are you suggesting we shouldn't use fuzzy matching on auto import? or wait for more characters?

[mjbvz]

The number of actions returned is overwhelming (and none of them are what I want) which makes the UX pretty poor

Could you filter by symbols that are a small edit distance from an actual symbol name instead of showing every possible match? Right now it feels like the quick fixes are being used more like how IntelliSense suggestions work

[rchiodo]

Maybe two characters would be better. Don't show it for one.

@mjbvz what would the list look like if it was symbols that were a small distance apart? Everything in your list seems like it has something that starts with a lower case 'a'.

[mjbvz]

Quick fixes are meant to show high confidence fixes for problems in the code. If I want to import antigravity for instance, we should not teach users to type an and then trigger quick fixes to select import antigravity. Instead antigravity should simply show in the suggestions (and it correctly already does if you enable auto imports)

For the quick fixes, JS/TS only shows import quick fixes for exact symbol matches. We've never had complaints about this (again likely because auto import completions handle the fuzzy case) If this is too restrictive for some reason, you could return auto import quick fixes for symbols that a very short Levenshtein distance away from the target symbol. As in, like 1 or two missing/transposed characters

[heejaechang]

I guess we can move it to discussion and gather users' opinions on filtering logics. current logic is simple, max 2 char apart fuzzy matching. more inputs (the word auto import acts upon) given, smaller suggestions provided. filtering is predicable. more input will reduce suggestions, but it won't introduce new suggestions.

...

That being said, unfortunately, python is not designed ground up IDE in mind, so its IDE supports can't be as nice as such languages, such as TS, C# and more.

languages that are created with IDE support in mind usually encode some data in language syntax or configuration to make IDE support easier. python puts those data in documentation (and no one standard) where it is hard for machine to extract those data. probably that is the reason why there is so many AI projects to help python programming.

For an example, unlike typescript which has project.json which can let LS knows what dependencies a project directly has, and language syntax indicates what symbol is supposed to be a public api (importable). python doesn't have such thing. TS IDE can reduce among 1000 node modules installed which 10 node modules are the actual direct dependencies, among 10000 symbols from dependencies, 100 symbols are public apis.

but python IDE can't. so, we can either add heuristic to reduce suggestions (which always has its own throwback) or show all possibilities with a way to reduce them. for now, we took the latter approach. If there is better way, we are always open to a suggestion.

[mjbvz]

current logic is simple, max 2 char apart fuzzy matchin

It seems like it's comparing prefixes then because otherwise antigravity would not match a, right? Could you apply this to the entire symbol name instead?

The suggest UI in editors like VS Code is designed specifically for this type fuzzy, prefix based matching. The quick fix UI should not be used for this though and I'm concerned it will teach users to think of quick fixes as a smarter version of the suggestion widget

[luabud]

We could make this list more relevant by checking packages that are listed in requirements.txt and pyproject.toml

[mjbvz]

That sounds like it may help but please still consider filtering the list to only show high confidence fixes

[heejaechang]

let's discuss it on the discussion. the worry I have on making the match over the entire symbol is some users don't write out whole word (many times they don't remember whole symbol name). and the filtering is no longer predictable. as user writes more characters, new suggestions will be shown when invoked. making it a guessing game.

I think one way we can try before changing filtering logic is trying to reduce candidate modules using things lke requirements.txt or pyproject.toml

[heejaechang]

by the way, I just checked how many people turn auto import on. it is less than 8% of all users. we will probably consider changing default only if that number goes up way higher than 8%. you need to remember that not all pylance users are experts. a lot of them don't even understand why new import suddenly added at the top of the file.

[mjbvz]

8% of all users explicitly turning on a setting is actually pretty high! To do this they have to know that this setting exists in the first place and then do the work to turn it on

---

Interested to hear what limitations you've hit with VS Code's suggestion API / UI. We've never designed around static languages. It could be that we already have a solution for whatever the problem it is, or maybe we need to consider some small API tweaks to help you

Also even though I'm talking about VS Code, the code actions API (and recommendation on how the API should be used) is almost identical in the LSP. The rendering of code actions is broadly similar across editors too, especially between VS and VS Code

[heejaechang]

I don't think 8% is high enough to change default. people who don't like auto import also complain when we turn it on by default.

to do this they have to know that this setting exists in the first place and then do the work to turn it on

that goes same for people who want to turn it off. so, either side we choose one side won't be happy even though all they need is go to the setting and turn it off or on.

I believe the experience should be designed to improve user's productivity and experience rather than to follow host or tool's internal rules.

let's see what other users think.

[luabud]

I believe we all we want the best experience for users, we're just not on the same page as to what is the best in this case 😅 I agree with HeeJae's point of this being convenient especially for common packages that can be hard to spell, but I also agree what we're showing in the example Matt raised is a bit too much, and that impacts readability. If the list is too long, it's hard to call it a "quick" fix, given it can take time to find the right term, as there's no way to filter the options (you need to escape from the code actions list, then type more, then ctrl + . again). I think a good middle ground could be the suggestion of using fuzzy match with a max 2 chars difference against the entire symbol name, but of course, like HeeJae said, I'd love to hear what other users think about this too.

[heejaechang]

I think having 1 char and invoking add import and complaining it shows too many matches is a bit too artificial. no one either will do that, or if what they are trying to import is truly 1 char module name, then it will show up at the top.

I dont think it is a good idea to tailor user experience over artificial example.

for example, if we apply this fuzzy match with a max 2 chars difference against the entire symbol name to the repro steps shown in this issue, user won't get what user is looking for (antigrivity) at all, so the user has to write more and invoke add import again anyway, so not sure what UX it makes better.

in my opinion, it is actually worse experience, since now user has to know that even if antigrivity is not in the list now, if user wrote more words, it might show up or might not if user misspelled more than 2 letters.

I am +1 on finding better UX but I don't think showing less for the sake of just showing less is better UX than what we have now.

[mjbvz]

I think having 1 char and invoking add import and complaining it shows too many matches is a bit too artificial

I hit this writing real python code. If you have an unresolved name in a python file, you get a red squiggle. If you then place the cursor on the red squiggle, you get a lightbulb suggesting that a fix is available. Triggering quick fixes often then shows an overwhelming list of suggestions. This list is so long that I can't find the handful of actual useful suggestions in the list

You get similar behavior for common variable names such as name:

Or date:

[heejaechang]

currently, I believe, we just sort the module name to make the list predictable with the exact match at the top. we can make the sorting a bit more sophisticated. like exact match at the top, and sorting with the module name length and then sorting module name within same length. that will solve I think most of your problems since regardless of the size of the list, what you wants will be at the top.

[mjbvz]

No, I only want to see those items. Showing such a long list is overwhelming even if the most relevant suggestions are at the top. Quick fixes should include fixes we are relatively confident in, not all possible fixes

[heejaechang]

well. unfortunately, list can still be long whatever we do. there can be 1 million symbols with name and we show 1 million of them.
so, saying exact match will reduce size of the list to small is not correct.

[mjbvz]

Yes but realistically you then can either place an upper limit the number of fixes returned and/or further filter the list to exclude ones that aren't relevant (for example, filtering based on the requirements file)

[heejaechang]

sure. I am +1 on reducing number of suggestions or improving how we show suggestions (ordering or others) to users. just not +1 on reducing it to exact match. many users don't remember exact name of long name symbols, for example, widgets or components in django. in my opinion, we shouldn't force them to use auto import when they want add import instead.

[brettcannon]

I think the key thing to consider for this situation is what would happen if there were two other language servers came in with the same number of entries for the same thing and had them placed before Pylance's? Would that facilitate finding the appropriate answer that Pylance may have, or would it hinder the user's search for the right answer?

After discussing this issue on the VS Code side we realized we need to update the extension guidelines to mention suggesting having more like 10 or less suggestions and to use heuristics when necessary to shorten the list as there isn't any public guidance on this topic.

If you're starting from a data-starved scenario like a single letter you could scrape GitHub to calculate import frequencies since chances are that aligns with what people want. We also realized people could lean on something like Copilot to help figure out what they may want when starting with less information.

[heejaechang]

I am +1 on reducing number of entries. reducing it to 10 or less sounds good, but not sure how we can do that for now. but open to any suggestions like calling copilot or some other AI services to reduce list. but that being said, the list doesn't only include symbols from third party packages, it also includes symbols from user own code in the workspace. so, there are some cases even copilot won't be able to help.

[mjbvz]

@brettcannon Design guidelines for code action likely belongs in the LSP spec, since I believe this is what Pylance targets

Here's what the LSP spec on code action requests already says:

The code action request is sent from the client to the server to compute commands for a given text document and range. These commands are typically code fixes to either fix problems or to beautify/refactor code.

We try not to make our API guidance too proscriptive so as not to limit creative new uses cases. However the relevant points for this example are:

• The code action list is meant to contain only relevant actions (we leave it up to languages to determine what counts as relevant, but in general after a certain point, increasing the number of returned code actions will make the entire list less relevant to the user)
• The code action list is not filterable (unlike the suggest API, it does not talk about filtering or highlighting support)
• For the LSP in general, use the API best suited to the use case. In this case, showing a long list of possibilities that inserts the rest of symbol name is a better fit for the suggest API

[heejaechang]

if having long list is really so much a bothering point for you, then one way to fix is vscode/LSP providing a way to group code fix suggestions like how VS does. so we can surface something like exact matches at the top and put other ones in nested menu.

[heejaechang]

here is how VS does it when there are a lot of fix to offers.

[heejaechang]

it basically puts the one user will most likely select at the top and put misc ones under nested menu so user can get less visual noise but still get all choices if they want/need.

[mjbvz]

Perhaps but until such functionality is added to the LSP, we have to design for the existing APIs + UIs

I'm also not convinced that showing lots of highly nested groups would be much of an improvement in this case. You'd still get a bunch of non-relevant fixes, now just under submenus instead of at the top level

[heejaechang]

I'm also not convinced that showing lots of highly nested groups would be much of an improvement in this case. You'd still get a bunch of non-relevant fixes, now just under submenus instead of at the top level

whether something is relevant or not is subjective. for me, long list doesn't bother as much, what bother me is if it doesn't have one that I want. and it don't have predictable behavior (filtering, sorting) that let me to find one I need.

by the way, your example above, you can't say the list is irrelevant to all peoples since you don't know what other people actually looking for with a. the list doesn't have any item that is 100% irrelevant from a already.

also, I believe VS added that submenu from user requests. if It helped VS users, then I am pretty sure it will help vscode users as well.

Perhaps but until such functionality is added to the LSP, we have to design for the existing APIs + UIs

my opinion is opposite. for me, vscode is a tool, letting people finish their work is much higher priority than neat UI. We should move to short top list once platform supports it. especially since VS already supports it. if LSP adds that ability, VS can already improve UX without scarifying any functionality.

also, I believe assuming suggested fix list can always be small is already proved wrong from other LS and IDE supporting code action or similar features.

I remember C# specifically adding Supress or Configure issues in code action with all possible routes because that was best user experiences on configuring fixes compared to asking users to edit editorconfig file or change options from option pages. it keeps users in code and change behavior at the spot when needed compared to moving to other context and losing engagement.

[ssbarnea]

I am perplexed about the lack of learning ability in QuickFix imports. While you might import for stdlib or same place for hundreds of times, it still fails to learn from previous actions and gives an alphabetically listed list of... mistakes.

One of the most annoying suggestions being on Mapping and Sequence, where it never displays typings as the 1st options. stdlib should have being always be prioritized, but is not....

[heejaechang]

@ssbarnea, If you open an issue describing behavior you want more in detail, we can certainly discuss about it. we recently made a change on sorting - #3760 - on completion.

I think we didn't make the same ordering on code action. but we can certainly do that if more people want that.

[DanielRosenwasser]

One of the suggestions I had in #4968 was to leverage nesting in the quick fix menu.

Add "import time"
Add "from datetime import time"
Add "from time import time"
See other possible imports for "time"...
├── Add "import timeit"
├── Add "from asynctimeout import timeout_at"
├── Add "from asynctimeout import timeout"
└── ...

That said, I do agree with the sentiment at #3873 (comment)

[heejaechang]

yes, that would be my preference as well. and we can leverage it on other fixes as well such as providing multiple ways to fix an issue (like how C# does in VS) but unfortunately, vscode or lsp doesn't support nested code actions. and it sounds like they explicitly doesn't want to support it.

[justschen]

Revisiting this since some users have been running into it and it hangs with the excessive code actions (microsoft/vscode#196857) . A quick solution for now is to limit the maximum number of code actions, which is being done in this PR. The top 50 non-AI code actions will be shown, with AI quick fixes at the bottom of the list. Code actions are already sorted prior to this slice so there shouldn't be any issues with more exact matches than the one provided in the issue above.

@heejaechang do you have any thoughts on this.

[heejaechang]

code action is cancellable and run in server process, so not sure how code action blocked UI as long as cancellation is issued properly. or are you saying it is a hang on rendering the code action list? in that case, issue seems on rendering.

50 seems big enough number but that said, I believe proper fix is providing extensions nested code actions apis like how VS solved the issue. just randomly limiting number of code actions will make code action a lot less useful in my opinion.

for me, the root issue of this seems code action being flat list and only using sorting as a way to show priority. assuming diagnostic will only have 1 proper fix is very wrong assumption in my opinion.

for example, 1 diagnostic say unknown variable can have multiple different way to solve it such as 1. add import if there are matches (even if it is exact matches), 2. generate type, 3. generate function, 4. move variable 5. ignore diagnostic and refactorings if there is any.

and each kind of fix can have multiple suggestions for it. for add import, it can show multiple matches. for generate type, it shows where to create the type. in current file, new file and etc. same for generate function. move variable can list where to move the unknown variable. ignore diagnostic can show how to ignore the diagnostic.

that would already pass at least 10 code actions mark.

and this is not theoretical, since this is actually what C# shows for unknown identifier.

https://user-images.githubusercontent.com/1333179/216735847-09252353-7404-48ac-a185-e0d3aaca7679.png

anyway, we are planning to test with nested code actions in VS (since their VS LSP extension support nested code actions) to gather some telemetry.

[heejaechang]

by the way, in VS where nested code actions are supported, how C# work is and how we will do is showing just 1 best guess at the top code action list. for example, showing only Mapping from typing at the top, and any other matches for Mapping in the nested menu such as one from collections.abc and etc. same goes for other type of code actions such as generate type/function and etc.

this will make list of code actions at the top very small but doesn't remove usability too much since user can access all other options from nested menu.

also, if there are too many top level code actions still, VS automatically does overflow menu to put lower priority code actions in sub menu automatically. for example, since C# support custom code actions from users, even C# team don't know how many code action (fix), refactoring will be available at the position.

[luabud]

We're going to address #5226 as a solution to this problem

[luabud]

Closing as this has been addressed by #5226