Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Complete SQLAlchemy inline pep484 typing #6810

Open
4 of 38 tasks
CaselIT opened this issue Jul 24, 2021 · 31 comments
Open
4 of 38 tasks

Complete SQLAlchemy inline pep484 typing #6810

CaselIT opened this issue Jul 24, 2021 · 31 comments
Assignees
Labels
big job a major change that requires a lot of knowledge and work code review in progress code has been provided that's in review as PR and/or gerrit Epic Use this tag to define an Issue as an Epic that contains multiple issues for a targeted help wanted Extra attention is needed PRs (with tests!) welcome a fix or feature which is appropriate to be implemented by volunteers time consuming / tedious an issue that will take a long time to complete, even though it may be "easy" typing pep -484 typing issues. independent of "mypy"
Milestone

Comments

@CaselIT
Copy link
Member

CaselIT commented Jul 24, 2021

Status as of 2023-02

Most of the library is typed, but some modules/packages that are still work in progress.

Modules / packages that still require work

  • sqlalchemy.dialects popular api
    • mssql - missing most/all types
    • mysql - missing most/all types
    • oracle - missing most/all types
    • postgresql - missing most/all types
    • sqlite - missing most/all types
  • sqlalchemy.engine
    • cursor - passes omitting non-typed functions - internal module
    • default - passes omitting non-typed functions - internal module
  • sqlalchemy.ext
    • declarative.extentions - missing most/all types - legacy
    • backed - missing most/all types - legacy
    • compiler - missing most/all types - popular api
    • indexable - missing most/all types
    • instrumentation - missing most/all types
    • orderinglist - missing most/all types
    • serializer - missing most/all types - legacy
  • sqlalchemy.orm
    • bulk_persistence - missing most types - internal module
    • collections - missing most types - internal module
    • context - missing most types - internal module
    • dependency - missing most types - internal module
    • dynamic - missing most types - internal module
    • evaluator - missing most/all types - internal module
    • instrumentation - missing some types - internal module
    • loading - missing most types - internal module
    • persistence - missing most types - internal module
    • strategies - missing most types - internal module
    • strategies - missing most types - internal module
    • unitofwork - missing most types - internal module
    • util - missing most types - internal module
    • writeonly - missing most types - internal module
  • sqlalchemy.sql
    • base - missing most types - internal module
    • compiler - missing most types - internal module
    • crud - missing most types - internal module
    • ddl - missing most types - internal module
    • functions - missing most types - internal module
    • lambdas - missing most types - internal module
    • naming - missing most types - internal module
    • roles - missing most types - internal module
    • traversals - missing most types - internal module
    • util - missing most types - internal module
@CaselIT CaselIT added big job a major change that requires a lot of knowledge and work time consuming / tedious an issue that will take a long time to complete, even though it may be "easy" pep484 Epic Use this tag to define an Issue as an Epic that contains multiple issues for a targeted SQLA mypy plugin mypy plugin issues only. general pep-484 issues should be "typing" labels Jul 24, 2021
@CaselIT CaselIT added this to the 2.0 milestone Jul 24, 2021
@CaselIT
Copy link
Member Author

CaselIT commented Nov 22, 2021

We should try https://github.com/ambv/retype/

@CaselIT CaselIT added this to 2.0 deliverables backlog in sqlalchemy 2 via automation Nov 22, 2021
@CaselIT CaselIT added the help wanted Extra attention is needed label Dec 19, 2021
@sqla-tester
Copy link
Collaborator

Federico Caselli referenced this issue:

Properly type _generative https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/3423

sqlalchemy-bot pushed a commit that referenced this issue Dec 31, 2021
Good new is that pylance likes it and copies over the
singature and everything.
Bad news is that mypy does not support this yet python/mypy#8645
Other minor bad news is that non_generative is not typed. I've tried using a protocol
like the one in the comment but the signature is not ported over by pylance, so it's
probably best to just live without it to have the correct signature.

notes from mike:  these three decorators are at the core of getting
the library to be typed, more good news is that pylance will
do all the things we like re: public_factory, see
microsoft/pyright#2758 (comment)
.

For @_generative, we will likely move to using pep 673 once mypy
supports it which may be soon.  but overall having the explicit
"return self" in the methods, while a little inconvenient, makes
the typing more straightforward and locally present in the files
rather than being decided at a distance.   having "return self"
present, or not, both have problems, so maybe we will be able
to change it again if things change as far as decorator support.
As it is, I feel like we are barely squeaking by with our decorators,
the typing is already pretty out there.

Change-Id: Ic77e13fc861def76a5925331df85c0aa48d77807
References: #6810
@sqla-tester
Copy link
Collaborator

Mike Bayer has proposed a fix for this issue in the main branch:

initial reorganize for static typing https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/3447

@zzzeek
Copy link
Member

zzzeek commented Jan 9, 2022

related issues:

#7535
#7519
#7562
#7555

@zzzeek
Copy link
Member

zzzeek commented Jan 9, 2022

also there's the issue of replacing existing sqlalchemy-stubs packages in type checkers and it's suggested we use a "positive" masking approach, as a "negative" uninstall approach is not supported by type checkers that bundle the stubs: microsoft/pylance-release#840 (comment)

@sqla-tester
Copy link
Collaborator

Mike Bayer has proposed a fix for this issue in the main branch:

WIP for ORM typing https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/3495

@zzzeek zzzeek added typing pep -484 typing issues. independent of "mypy" and removed help wanted Extra attention is needed labels Jan 9, 2022
@zzzeek zzzeek self-assigned this Jan 9, 2022
@CaselIT CaselIT removed the pep484 label Jan 10, 2022
@zzzeek
Copy link
Member

zzzeek commented Jan 10, 2022

current goals are:

  1. get https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/3495, based on https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/3447, working. POC here is that I would like to include updating the mypy plugin and removing the dependency on sqlalchemy2-stubs, w/ mypy tests passing. that will be the proof of concept for 3495, which will also validate 3447.
  2. then merge 3447 and 3495
  3. after those two are in, that's the "hard part" of typing, that is, the column expression / operator / ORM mapping stuff. then we can do a package-by-package or module-by-module approach in separate gerrits
  4. separately, when doing the typing I hit a lot of blocks which are due to the fact that we use metaclasses for "visitable" items as well as some other things. I want to see how far we can use __subclass_init__ to get rid of all metaclasses, at the very least the metaclasses used internally by "visitable" as well as functions.py etc. The DeclarativeMeta metaclass might be a different situation if this is going to break backwards compatibility.

@CaselIT
Copy link
Member Author

CaselIT commented Jan 10, 2022

Step 4. was something that I meant to suggest since we now are guaranteed to always have at least py3.6

does it make sense as a separate change?

@zzzeek
Copy link
Member

zzzeek commented Jan 10, 2022

im going to try it now as its own thing, not including declarative, which probably also needs to be its own thing

@zzzeek
Copy link
Member

zzzeek commented Jan 10, 2022

I think we might want to skip declarative as the DeclarativeMeta class also intercepts class-level setattr and delattr events. in that sense we are still using metaclass-only features.

i have all the internal stuff off of metaclasses with one small exception in the ORM, will keep it at that.

@sqla-tester
Copy link
Collaborator

Mike Bayer referenced this issue:

remove internal use of metaclasses https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/3499

sqlalchemy-bot pushed a commit that referenced this issue Jan 11, 2022
All but one metaclass used internally can now
be replaced using __init_subclass__().  Within this
patch we remove:

* events._EventMeta
* sql.visitors.TraversibleType
* sql.visitors.InternalTraversibleType
* testing.fixtures.FindFixture
* testing.fixtures.FindFixtureDeclarative
* langhelpers.EnsureKWArgType
* sql.functions._GenericMeta
* sql.type_api.VisitableCheckKWArg (was a mixture of TraversibleType
  and EnsureKWArgType)

The remaining internal class is MetaOptions used by the
sql.Options object which is in turn currently mostly for
ORM internal use, as this type implements class level overrides
for the ``+`` operator.

For declarative, removing DeclarativeMeta in place of
an `__init_subclass__()` class would not be fully feasible as
it would break backwards compatibility with applications that
refer to this class explicitly, but also DeclarativeMeta intercepts
class-level attribute set and delete operations which is a widely
used pattern.   An option for declarative base to use
`__init_subclass__()` should be provided but this is out of
scope for this particular change.

Change-Id: I8aa898c7ab59d887739037d34b1cbab36521ab78
References: #6810
@sqla-tester
Copy link
Collaborator

Mike Bayer referenced this issue:

Initial ORM typing layout https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/3495

sqlalchemy 2 automation moved this from 2.0 deliverables backlog to 2.0 deliverables done Jan 14, 2022
@CaselIT CaselIT reopened this Jan 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
big job a major change that requires a lot of knowledge and work code review in progress code has been provided that's in review as PR and/or gerrit Epic Use this tag to define an Issue as an Epic that contains multiple issues for a targeted help wanted Extra attention is needed PRs (with tests!) welcome a fix or feature which is appropriate to be implemented by volunteers time consuming / tedious an issue that will take a long time to complete, even though it may be "easy" typing pep -484 typing issues. independent of "mypy"
Projects
No open projects
sqlalchemy 2
  
2.0 deliverables done
Development

No branches or pull requests

5 participants