Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Auto-completion when instantiating BaseModel objects #650

Closed
marlonjan opened this issue Jul 10, 2019 · 42 comments
Closed

Auto-completion when instantiating BaseModel objects #650

marlonjan opened this issue Jul 10, 2019 · 42 comments

Comments

@marlonjan
Copy link

marlonjan commented Jul 10, 2019

Feature Request

Hi! Currently auto-completion does not work well when calling the initializer of classes that inherit from pydantic.BaseModel, at least in IntelliJ / PyCharm.

Are there any plans to improve that? Auto-completion works nicely with dataclasses from the python 3.7 standard library. Same applies to highlighting type mismatches or missing arguments in the call. Here is a comparison of the behavior in IntelliJ:

image

image

Code example to reproduce the screenshots:

from pydantic import BaseModel
from dataclasses import dataclass


class User(BaseModel):
    id: int
    name: str
    age: float


@dataclass
class UserDataclass:
    id: int
    name: str
    age: float


# no auto-completion
user = User()

# auto-completion
user_dataclass = UserDataclass()

Thanks for creating pydantic! Best regards,
Marlon

@timonbimon
Copy link
Contributor

adding auto-completion support would be huge!

@dmontagu
Copy link
Contributor

I would also love to see this.

attrs seems to be able to (very flexibly) generate a mypy-compatible type-hinted __init__, so I think this might be possible, at least in theory.

However, one potential issue is that parsing is performed during initialization, so you may purposely want to avoid using annotated types as type hints on the initialization kwargs. I do think it would be nice to have a type-hinted __init__ (or other method, I suppose) for creating model instances where parsing is known to be unnecessary, even if it means that a separate method had to be used for type-hint-safe parsing (I expect this could have some performance benefits as well).

If I recall correctly, there has been some discussion of replacing the use of BaseModel.__init__ with BaseModel.parse_object (or something similar) for parsing; I would expect that would be necessary for compatibility with a type-hinted __init__. @samuelcolvin I would be interested to get your take on whether that is realistic.

@samuelcolvin
Copy link
Member

I would love this too.

I don't know how pycharm builds suggestions, does it use mypy or it's own parsing?

A mypy plugin as suggested in #460 would fix this case in mypy, but would it help in pycharm?

Maybe @pauleveritt who I think uses pydantic and works for pycharm might know?

I'm at pycharm europe now which is sponsored by pycharm so maybe I can find a real human being to ask.

@dmontagu
Copy link
Contributor

dmontagu commented Jul 10, 2019

Okay, I dug into this just a little:

  1. it looks like mypy ships with an attrs-specific plugin to make the __init__ checking work: https://github.com/python/mypy/blob/master/mypy/plugins/attrs.py
  2. I don't have a reference on me, but I've looked into this in the past, and if I recall correctly, PyCharm has a custom extension to ensure attrs __init__s look right. (Presumably this is also how they are handling dataclass). So it might be hard to get this support in a native way (though it would be great to hear from JetBrains if they had any suggestions).

Separately, I think it might be possible to achieve this by "tricking" the IDE in a variety of ways. I played with various approaches involving TYPE_CHECKING, but I think the one I currently like best is the following:
Screen Shot 2019-07-10 at 4 16 34 AM

This behaves as though undecorated at runtime, but seems to behave properly with both PyCharm and mypy (retaining all desired autocomplete behavior for the BaseModel class). @marlonjan and @timonbimon you might find this useful in the short term?

@samuelcolvin I don't know how great this trick is as a long term solution, but for the sake of "paving the way" for such a capability it would probably makes sense to push in the direction of changing the use of the __init__ method so it is never expected to be called with incorrect types (i.e., for parsing). Maybe this could be a 1.0 thing?

@samuelcolvin
Copy link
Member

Just spoken to pycharm in person, short answer is that we would need to build a pycharm extension in Java or kotlin to do this, attrs or dataclass code is open source and would be a good starting point.

@koxudaxi
Copy link
Contributor

I want to support auto-completion on PyCharm.
@samuelcolvin
Thank you for asking the person.

I have searched issues and source code for attrs and dataclass.
The issue is to support auto-completion for attrs.
https://youtrack.jetbrains.com/issue/PY-26354

Also, we can find related commit(source code) by the issue number(PY-26354) in intellij-community repository.
https://github.com/JetBrains/intellij-community/search?o=asc&p=1&q=PY-26354&s=author-date&type=Commits

JetBrains developer implemented auto-completion for attrs and dataclass by request.
However, I don't know we should request to the developers.

if you already know the information, then I'm sorry.

@gmetzker
Copy link

This would be a great feature / plugin!

@samuelcolvin
Copy link
Member

Jet brains people were very nice but said pydantic is not popular enough for them to build the plugin themselves, however they will help if we have any issues.

@dmontagu dmontagu mentioned this issue Jul 10, 2019
@pauleveritt
Copy link

Yes, I'm watching on this ticket. I'm here at EuroPython with the PyCharm team. If anybody has any questions about compiling PyCharm from GitHub, making a plugin, or what the specifics about a pydantic plugin, let me know.

@koxudaxi
Copy link
Contributor

@pauleveritt
Thank you for watching the issue.
I have a question, Should we make a plugin or patching Pycharm?
If we can merge the feature on PyCharm then, I think that it is the best way.
However, I don't know you and your company want to merge the feature on the mainstream.

Btw, I have patched PyCharm and build it for auto-completion.
2019-07-12 14 58 44

It's an experimental challenge yet. Console show errors :(

@dmontagu
Copy link
Contributor

@koxudaxi Would you be willing to share your implementation?

@pauleveritt
Copy link

@koxudaxi sorry for the delay in replying, I'm working the PyCharm booth at EuroPython. Kudos for getting as far as you have! I can probably get someone from the PyCharm team to look at your code if you can put it in a gist or in a fork.

As you might imagine, PyCharm tries to be careful about the things it commits to supporting. Over the years it adds up, and we have to keep supporting things even when most people stop using them. So it's better to do it as a plugin and, if it becomes popular, we could discuss a merge.

@koxudaxi
Copy link
Contributor

@koxudaxi Would you be willing to share your implementation?

@dmontagu sorry, I have forgotten it.
This commit is a prototype. if we hope to publish it then we need to implement more code.
I could add pydantic.dataclass easy. but, we will need to spend time to implement BaseModel which is different from other dataclasses like attrs.

koxudaxi/intellij-community@c654365

@koxudaxi
Copy link
Contributor

@pauleveritt
Thank you for your explanation. It's all clear now.
I try to create a plugin :)

Also, I use Jetbrains products every day, which increase development speed. If I contribute to your community, I'm happy. Thank you.

@marlonjan
Copy link
Author

Thanks to all of you for taking a look at this!!

@pauleveritt
Copy link

Thank you @koxudaxi and if you want me to get someone from the team to review, let me know.

I personally am very, very happy...VERY happy...to see you work on this.

@koxudaxi
Copy link
Contributor

@pauleveritt
Thank you for your kindness. If we need some help, then we ask it to you.

@dmontagu
Copy link
Contributor

dmontagu commented Aug 6, 2019

@koxudaxi any chance you could explain how to build pycharm to make use of your modification? (Or point me in the direction of a guide?) I've never worked with custom pycharm plugins. I see it is in the README.md of your linked repo. Do you know if it is hard to use the plugin with the pro version of PyCharm? (It looks like building is against the community version; not sure what it looks like to integrate with pro.)

Related, I've made some progress on a mypy plugin, linked on this issue: #460. It definitely doesn't behave properly under all circumstances yet, but it at least works for me to type-check BaseModel.__init__ for all of the models I use. (I probably tend to follow some conventions that keep it away from the as-yet unhandled edge cases).

@koxudaxi
Copy link
Contributor

koxudaxi commented Aug 7, 2019

@dmontagu

Do you know if it is hard to use the plugin with the pro version of PyCharm? (It looks like building is against the community version; not sure what it looks like to integrate with pro.)

I use the plugin in the pro version.
However, the plugin is on a very experimental phase.
It happens a lot of errors. @MrMrRobat have reported (koxudaxi/pydantic-pycharm-plugin#1)

It looks like building is against the community version; not sure what it looks like to integrate with pro.

Sorry, I don't know it. I must learn how to build a plugin for each edition.
I will try to fix the plugin this week.

I have checked your mypy-plugin, I feel a great start. :)
If we cant use mypy-plugin to do auto-complete and check types in Pycharm, then I think the best way.
We can maintain the plugin easier than Kotlin-plugin.

@dmontagu
Copy link
Contributor

dmontagu commented Aug 7, 2019

Yeah, I was pleasantly surprised how easy it was to work with once I got started.

Related, there also appears to be some good infrastructure for calling mypy runs from within python, which I think would enable us to more easily/incrementally add mypy tests (in particular, expected fails) than the current system.

I would be surprised if the mypy plugin worked directly with PyCharm, so it would probably still be nice to build an autocomplete plugin for PyCharm.

@koxudaxi if you can think of any specific requests for the mypy plugin beyond type checking the __init__ call, please post them to #460 and I'll try to add them.

@koxudaxi
Copy link
Contributor

koxudaxi commented Aug 7, 2019

@dmontagu
I have fixed easy bugs on my pycharm-plugin. I continue developing the plugin.
OK, I will post some request or question on the issue.
Thank you.

@samuelcolvin
Copy link
Member

great work @koxudaxi.

Feel free to ping me here if there's anything I can do to help make plugin development easier in pydantic.

@koxudaxi
Copy link
Contributor

The PyCharm plugin for Pydantic is now available on JetBrains Plugins Repository.
The Plugin can be installed from Marketplace (PyCharm's Preferences -> Plugin -> Marketplace)

The Plugin page is https://plugins.jetbrains.com/plugin/12861-pydantic

Also, I am developing the plugin on this repository https://github.com/koxudaxi/pydantic-pycharm-plugin
If you have any opinion, Would you please leave feedback?
I'm waiting for your bug reports and feature requests, PRs.

@samuelcolvin
Copy link
Member

Fantastic, well done.

I've installed it and at first glance it seems to do everything I need.

@samuelcolvin
Copy link
Member

samuelcolvin commented Aug 15, 2019

@koxudaxi would you consider submitting a PR to add this to the documentation?

@koxudaxi
Copy link
Contributor

@samuelcolvin
I am happy seeing your good response 😄

OK, I will create the PR.

@davideasaf
Copy link

Hey everyone,

Does anyone know if a similar solution exists within VsCode? Auto-completion when instantiating BaseModel objects would be awesome there as well.

@RafalSkolasinski
Copy link

RafalSkolasinski commented Jul 19, 2020

I just hit the completion problem when using Sublime with LSP and python-language-server.

Took me a while to figure out that completion for methods in class inheriting from BaseModel are not working and it's not something wrong with my code ;)

Any good workarounds until this is resolved?

pyls logs if helpful

2020-07-19 13:54:40,852 UTC - ERROR - pyls_jsonrpc.endpoint - Failed to handle request 6
Traceback (most recent call last):
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pyls_jsonrpc/endpoint.py", line 113, in consume
    self._handle_request(message['id'], message['method'], message.get('params'))
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pyls_jsonrpc/endpoint.py", line 182, in _handle_request
    handler_result = handler(params)
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pyls_jsonrpc/dispatchers.py", line 23, in handler
    return method(**(params or {}))
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pyls/python_ls.py", line 322, in m_text_document__completion
    return self.completions(textDocument['uri'], position)
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pyls/python_ls.py", line 240, in completions
    completions = self._hook('pyls_completions', doc_uri, position=position)
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pyls/python_ls.py", line 156, in _hook
    return hook_handlers(config=self.config, workspace=workspace, document=doc, **kwargs)
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pluggy/manager.py", line 337, in traced_hookexec
    return outcome.get_result()
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pluggy/callers.py", line 52, in from_call
    result = func()
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pluggy/manager.py", line 335, in <lambda>
    outcome = _Result.from_call(lambda: oldcall(hook, hook_impls, kwargs))
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pyls/plugins/jedi_completion.py", line 74, in pyls_completions
    ready_completions = [
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pyls/plugins/jedi_completion.py", line 75, in <listcomp>
    _format_completion(c, include_params)
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/pyls/plugins/jedi_completion.py", line 145, in _format_completion
    'documentation': _utils.format_docstring(d.docstring()),
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/jedi/api/classes.py", line 719, in docstring
    return super(Completion, self).docstring(raw=raw, fast=fast)
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/jedi/api/classes.py", line 300, in docstring
    doc = self._get_docstring()
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/jedi/api/classes.py", line 728, in _get_docstring
    return super(Completion, self)._get_docstring()
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/jedi/api/classes.py", line 311, in _get_docstring
    return self._name.py__doc__()
  File "/home/rskolasinski/.local/lib/python3.8/site-packages/jedi/inference/compiled/value.py", line 329, in py__doc__
    value, = self.infer()
ValueError: not enough values to unpack (expected 1, got 0)

@talarari
Copy link

talarari commented Oct 15, 2020

i'm using vscode and would really like to have auto completion for creating models.
static type checkers (pyright for example) have no way of knowing what the arguments for init are, but they do work well with dataclasses

would it make sense to have a class inherit from BaseModel and be decorated with dataclass ?

from dataclasses import dataclass
from pydantic import BaseModel

@dataclass
class User(BaseModel):
    id: int
    name: str
    age: float

@Bobronium
Copy link
Contributor

Bobronium commented Oct 15, 2020

would it make sense to have a class inherit from BaseModel and be decorated with dataclass ?

I doubt that code above would work properly, but what you can try is to make fake dataclass decorator and trick type checker to believe that it's decorated with real dataclass:

from typing import TYPE_CHECKING

from pydantic import BaseModel

if TYPE_CHECKING:
    from dataclasses import dataclass
else:
    def dataclass(model):
        return model

@dataclass
class User(BaseModel):
    id: int
    name: str
    age: float

@talarari
Copy link

talarari commented Oct 16, 2020

this does work 👍
image

The thing is I don't want to trick my team mates when I trick the type checker, it will make it look like they are actually dataclasses...

i tried using just the pydantic dataclass but that gives this result:

from pydantic.dataclasses import dataclass

@dataclass
class User:
    id: int
    name: str
    age: float

image

could be great if the pydantic dataclass would do that maybe, and then i could just use `@dataclass from pydantic possibly (when fastapi fully supports it)

@talarari
Copy link

using @MrMrRobat advice i currently went with this workaround which seems to work with vscode and is not terribly misleading:

created an auto_complete_only.py helper and aliased @ dataclass to @ autocomplete

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from dataclasses import dataclass
else:

    def dataclass(model):
        return model


autocomplete = dataclass

image

@StephenBrown2
Copy link
Contributor

Could even be simpler (tho untested):

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from dataclasses import dataclass as autocomplete
else:
    def autocomplete(model):
        return model

🤷‍♂️

@JosXa
Copy link
Contributor

JosXa commented Dec 12, 2020

using @MrMrRobat advice i currently went with this workaround which seems to work with vscode and is not terribly misleading:

created an auto_complete_only.py helper and aliased @ dataclass to @ autocomplete

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from dataclasses import dataclass
else:

    def dataclass(model):
        return model


autocomplete = dataclass

image

image

Oh god, you must be kidding me. Pyright is so incredibly buggy...

@Bobronium
Copy link
Contributor

Bobronium commented Dec 12, 2020

@josx, as I understand, it's more a cosmetic issue, but you can try this workaround to use workaround above 🙃 without warnings:

from typing import TYPE_CHECKING

# trick PyRight into thinking that it might be both False or True
RUNTIME = not TYPE_CHECKING

if RUNTIME:
    def autocomplete(model):
        return model
else:
    from dataclasses import dataclass as autocomplete

I didn't test it, but it should work.

@samuelcolvin
Copy link
Member

I'm closing this as pycharm has an excellent plugin built by @koxudaxi and vscode support is discussed in microsoft/python-language-server#1898

It sounds like any further solution either requires changes to the language servers or third party plugins.

If there are any tweaks pydantic can make to make the jobs of language servers easier, please create a new issue.

If anyone succeeds in building a useful plugin, please create a PR to update the docs.

This was referenced Dec 28, 2020
@adamal
Copy link

adamal commented Jan 7, 2021

Very nice to find that a workaround exists even if somewhat hacky.

I came up with this simple extension of the idea – here adapted for BaseModel.

With this, you can import BaseModel from here (say type_help.py) in stead of from pydantic, and if/when a proper solution becomes available, you can just switch your import references without removing a host of @autocomplete-annotations.

from typing import TYPE_CHECKING
from pydantic import BaseModel as _BaseModel

# trick PyRight into thinking that it might be both False or True
RUNTIME = not TYPE_CHECKING

if RUNTIME:
    def base_model(model):
        return model
else:
    from dataclasses import dataclass as base_model

@base_model
class BaseModel(_BaseModel): ...

@podj
Copy link

podj commented May 27, 2021

@koxudaxi Hey man, great job!
Let me know if you need any help with the maintenance! You've saved lives, my friend!

@amir20
Copy link

amir20 commented Dec 27, 2021

For anybody following this issue, the fix has been merged and available in version 1.9.0a2.

@Tomperez98
Copy link

When will v 1.9.0 be available?

@samuelcolvin
Copy link
Member

Next few days.

@masreplay
Copy link

masreplay commented Jan 5, 2022

@samuelcolvin now I'm using SQLModel and autocompletion it's not with Pycharm work
@tiangolo

mcgfeller added a commit to mcgfeller/DDH that referenced this issue Dec 25, 2022
workaround to make pyright check Pydantic code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests