Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

from sqlalchemy import create_engine import error #6069

Closed
alexswo opened this issue Mar 17, 2021 · 23 comments
Closed

from sqlalchemy import create_engine import error #6069

alexswo opened this issue Mar 17, 2021 · 23 comments
Labels
awaiting info waiting for the submitter to give more information bug Something isn't working regression something worked and was broken by a change sql
Milestone

Comments

@alexswo
Copy link

alexswo commented Mar 17, 2021

Describe the bug
Getting the following when importing 1.4.0 sqlalchemy's create_engine method:

Python 2.7.5 (default, Aug  7 2019, 00:51:29)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from sqlalchemy import create_engine
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/__init__.py", line 9, in <module>
    from .engine import create_engine
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/__init__.py", line 18, in <module>
    from . import events
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/events.py", line 9, in <module>
    from .base import Engine
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 12, in <module>
    from .interfaces import Connectable
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/interfaces.py", line 11, in <module>
    from ..sql.compiler import Compiled  # noqa
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/__init__.py", line 13, in <module>
    from .expression import Alias
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/expression.py", line 144, in <module>
    from .lambdas import lambda_stmt
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/lambdas.py", line 1067
    exec(code, vars_, vars_)
SyntaxError: unqualified exec is not allowed in function '_rewrite_code_obj' it contains a nested function with free variables
@alexswo alexswo added the requires triage New issue that requires categorization label Mar 17, 2021
@zzzeek
Copy link
Member

zzzeek commented Mar 17, 2021

hi i need the exact Python interpreter version you are using

@zzzeek zzzeek added awaiting info waiting for the submitter to give more information bug Something isn't working sql and removed requires triage New issue that requires categorization labels Mar 17, 2021
@zzzeek
Copy link
Member

zzzeek commented Mar 17, 2021

also a reproduction case please

@zzzeek zzzeek added this to the 1.4.x milestone Mar 17, 2021
@zzzeek zzzeek added the regression something worked and was broken by a change label Mar 17, 2021
@alexswo
Copy link
Author

alexswo commented Mar 17, 2021

sorry.
updated the post with exact python version (2.7.5) and a use case

@zzzeek
Copy link
Member

zzzeek commented Mar 17, 2021

this might be a python versioning issue. any chance you can try a newer python 2.7? I have 2.7.18 here, for example.

@laozhu
Copy link

laozhu commented Mar 17, 2021

Hi @zzzeek
I have the similar importError problem with python3.9.1 and sqlalchemy 1.4.
BTW, with sqlalchemy 1.3.23 it's OK.

File "./services/main.py", line 4, in <module>
    from services.core.database import create_tables
  File "./services/core/database.py", line 16, in <module>
    database_master = Database(_settings.database_master_url)
  File "/Users/Ritchie/Developer/@easynm/cloud/services/.venv/lib/python3.9/site-packages/databases/core.py", line 66, in __init__
    backend_cls = import_from_string(backend_str)
  File "/Users/Ritchie/Developer/@easynm/cloud/services/.venv/lib/python3.9/site-packages/databases/importer.py", line 21, in import_from_string
    raise exc from None
  File "/Users/Ritchie/Developer/@easynm/cloud/services/.venv/lib/python3.9/site-packages/databases/importer.py", line 18, in import_from_string
    module = importlib.import_module(module_str)
  File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/Users/Ritchie/Developer/@easynm/cloud/services/.venv/lib/python3.9/site-packages/databases/backends/mysql.py", line 9, in <module>
    from sqlalchemy.engine.result import ResultMetaData, RowProxy
ImportError: cannot import name 'RowProxy' from 'sqlalchemy.engine.result' (/Users/Ritchie/Developer/@easynm/cloud/services/.venv/lib/python3.9/site-packages/sqlalchemy/engine/result.py)

@zzzeek
Copy link
Member

zzzeek commented Mar 17, 2021

Hi @zzzeek
I have the similar importError problem with python3.9.1 and sqlalchemy 1.4.
BTW, with sqlalchemy 1.3.23 it's OK.

you have a different problem. you seem to be using this package which is apparently not working for SQLAlchemy 1.4 . I would report this on their issue tracker at https://github.com/encode/databases .

@laozhu
Copy link

laozhu commented Mar 17, 2021

@zzzeek Thank you for reply, solved my problem 👍
I will stay with 1.3.23 until encode/databases#298 been solved.

@vphilippon
Copy link

From what I can see, the issue indeed only occurs with older py2.7 version. I tested with 1.4.0 and 1.4.1, same result.

Ex: Using Python 2.7.18:

$ docker run --rm -ti python:2.7 bash
root@12e12b250b9f:/# python -m pip install sqlalchemy
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting sqlalchemy
  Downloading SQLAlchemy-1.4.1-cp27-cp27mu-manylinux1_x86_64.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 3.5 MB/s
Requirement already satisfied: importlib-metadata; python_version < "3.8" in /usr/local/lib/python2.7/site-packages (from sqlalchemy) (1.6.0)
Requirement already satisfied: contextlib2; python_version < "3" in /usr/local/lib/python2.7/site-packages (from importlib-metadata; python_version < "3.8"->sqlalchemy) (0.6.0.post1)
Requirement already satisfied: pathlib2; python_version < "3" in /usr/local/lib/python2.7/site-packages (from importlib-metadata; python_version < "3.8"->sqlalchemy) (2.3.5)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python2.7/site-packages (from importlib-metadata; python_version < "3.8"->sqlalchemy) (1.2.0)
Requirement already satisfied: configparser>=3.5; python_version < "3" in /usr/local/lib/python2.7/site-packages (from importlib-metadata; python_version < "3.8"->sqlalchemy) (4.0.2)
Requirement already satisfied: six in /usr/local/lib/python2.7/site-packages (from pathlib2; python_version < "3"->importlib-metadata; python_version < "3.8"->sqlalchemy) (1.14.0)
Requirement already satisfied: scandir; python_version < "3.5" in /usr/local/lib/python2.7/site-packages (from pathlib2; python_version < "3"->importlib-metadata; python_version < "3.8"->sqlalchemy) (1.10.0)
Installing collected packages: sqlalchemy
Successfully installed sqlalchemy-1.4.1
WARNING: You are using pip version 20.0.2; however, version 20.3.4 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
root@12e12b250b9f:/# python
Python 2.7.18 (default, Apr 20 2020, 19:27:10) 
[GCC 8.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlalchemy
# No error or issue.

Now using Python 2.7.8 (the version on which the issue occured for me):

$ docker run --rm -ti python:2.7.8 bash
root@feb1ce332e69:/# python -m pip list
pip (1.5.6)
setuptools (7.0)
virtualenv (1.11.6)
wsgiref (0.1.2)
# Oh yikes, really old pip and setuptools in that docker image.
# That caused some issue at installation for me at first, so we need to update those.
# I do a first update to pip 10.0.1 which will properly handle Requires-Python metadata,
# I know that from memory/experience, and then I'll do another update to latest pip/setuptools/wheel.
root@feb1ce332e69:/# python -m pip install pip==10.0.1
Downloading/unpacking pip==10.0.1
  Downloading pip-10.0.1-py2.py3-none-any.whl (1.3MB): 1.3MB downloaded
Installing collected packages: pip
  Found existing installation: pip 1.5.6
    Uninstalling pip:
      Successfully uninstalled pip
Successfully installed pip
Cleaning up...
root@feb1ce332e69:/# python -m pip install -U pip setuptools wheel
/usr/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:339: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  SNIMissingWarning
/usr/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning
Collecting pip
/usr/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning
  Downloading https://files.pythonhosted.org/packages/27/79/8a850fe3496446ff0d584327ae44e7500daf6764ca1a382d2d02789accf7/pip-20.3.4-py2.py3-none-any.whl (1.5MB)
    100% |████████████████████████████████| 1.5MB 10.6MB/s
Collecting setuptools
  Downloading https://files.pythonhosted.org/packages/e1/b7/182161210a13158cd3ccc41ee19aadef54496b74f2817cc147006ec932b4/setuptools-44.1.1-py2.py3-none-any.whl (583kB)
    100% |████████████████████████████████| 583kB 13.4MB/s
Collecting wheel
  Downloading https://files.pythonhosted.org/packages/65/63/39d04c74222770ed1589c0eaba06c05891801219272420b40311cd60c880/wheel-0.36.2-py2.py3-none-any.whl
Installing collected packages: pip, setuptools, wheel
  Found existing installation: pip 10.0.1
    Uninstalling pip-10.0.1:
      Successfully uninstalled pip-10.0.1
  Found existing installation: setuptools 7.0
    Uninstalling setuptools-7.0:
      Successfully uninstalled setuptools-7.0
Successfully installed pip-20.3.4 setuptools-44.1.1 wheel-0.36.2
root@feb1ce332e69:/# python -m pip install sqlalchemy
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.      
/usr/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:424: SNIMissingWarning: An HTTPS request has been made, but the SNI (Server Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  SNIMissingWarning,
/usr/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:164: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning,
Collecting sqlalchemy
/usr/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:164: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning,
  Downloading SQLAlchemy-1.4.1-cp27-cp27m-manylinux1_x86_64.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 3.3 MB/s
Collecting importlib-metadata; python_version < "3.8"
  Downloading importlib_metadata-2.1.1-py2.py3-none-any.whl (10 kB)
Collecting contextlib2; python_version < "3"
  Downloading contextlib2-0.6.0.post1-py2.py3-none-any.whl (9.8 kB)
Collecting zipp>=0.5
  Downloading zipp-1.2.0-py2.py3-none-any.whl (4.8 kB)
Collecting configparser>=3.5; python_version < "3"
  Downloading configparser-4.0.2-py2.py3-none-any.whl (22 kB)
Collecting pathlib2; python_version < "3"
  Downloading pathlib2-2.3.5-py2.py3-none-any.whl (18 kB)
Collecting scandir; python_version < "3.5"
  Downloading scandir-1.10.0.tar.gz (33 kB)
Collecting six
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Building wheels for collected packages: scandir
  Building wheel for scandir (setup.py) ... done
  Created wheel for scandir: filename=scandir-1.10.0-cp27-cp27m-linux_x86_64.whl size=39138 sha256=6c35a5d95a7b3d8d1c81b2d8093bf5253a2d2785b7610ec9ef5b6b01224231db
  Stored in directory: /root/.cache/pip/wheels/58/2c/26/52406f7d1f19bcc47a6fbd1037a5f293492f5cf1d58c539edb
Successfully built scandir
Installing collected packages: contextlib2, zipp, configparser, scandir, six, pathlib2, importlib-metadata, sqlalchemy
Successfully installed configparser-4.0.2 contextlib2-0.6.0.post1 importlib-metadata-2.1.1 pathlib2-2.3.5 scandir-1.10.0 six-1.15.0 sqlalchemy-1.4.1 zipp-1.2.0
/usr/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:164: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning,
root@feb1ce332e69:/# python
Python 2.7.8 (default, Nov 26 2014, 22:28:51) 
[GCC 4.9.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlalchemy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/__init__.py", line 9, in <module>
    from .engine import create_engine
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/__init__.py", line 18, in <module>
    from . import events
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/events.py", line 9, in <module>
    from .base import Engine
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 12, in <module>
    from .interfaces import Connectable
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/interfaces.py", line 11, in <module>
    from ..sql.compiler import Compiled  # noqa
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/__init__.py", line 13, in <module>
    from .expression import Alias
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/expression.py", line 144, in <module>
    from .lambdas import lambda_stmt
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/lambdas.py", line 1067
    exec(code, vars_, vars_)
SyntaxError: unqualified exec is not allowed in function '_rewrite_code_obj' it contains a nested function with free variables
# Yup, here's the error.

@vphilippon
Copy link

I felt like I had seen a similar thing before causing issue with older py2.7.x versions, and that was in another project I was depending on. I found it, and maybe this'll help fixing the issue (I'm really not sure I get what's going exactly to be able to tell if the fix idea really applies here though, sorry if it doesn't).

https://github.com/pallets/werkzeug/pull/1545/files
(for pallets/werkzeug#1544)

@zzzeek
Copy link
Member

zzzeek commented Mar 17, 2021

yes there is mention of this issue also at plone/plone.protect#74 and https://bugs.python.org/issue21591 .

SQLAlchemy 1.4 was almost going to be Python 3 only, and the area in which this is occurring is a deep little spot where we are rewriting code objects, so I'm not sure what adjustment would fix it, if any, or maybe it's not that hard, but overall, if you look at places lke https://pypistats.org/packages/sqlalchemy we can see that Python 2 use / downloads are vastly less frequent than Python 3 downloads (in the 30K range, which is not nothing, but a lot less), this looks like a "wontfix" and at most we'd note it in the documentation. if you have an environment running on python < 2.7.9 or so, and newer Python 2.7s at least are not an option, that's an old environment, like an old Centos or whatever it is, it's to be expected that other packages need to stay on their older versions as well; it's not an env where new development should be occurring.

@zzzeek
Copy link
Member

zzzeek commented Mar 17, 2021

I felt like I had seen a similar thing before causing issue with older py2.7.x versions, and that was in another project I was depending on. I found it, and maybe this'll help fixing the issue (I'm really not sure I get what's going exactly to be able to tell if the fix idea really applies here though, sorry if it doesn't).

https://github.com/pallets/werkzeug/pull/1545/files
(for pallets/werkzeug#1544)

that actually looks like a really simple fix in fact, I thought the issue was how we were rewrting the code itself. if that's the case then there would be a bit of a path to doing this. but python 2.7.9 is really old. lots of packages are python 3 only now even.

@dwt
Copy link
Contributor

dwt commented Mar 22, 2021

I'd like to add that I see this issue on our build server (CentOS 7) that is still using Python 2.7.5. As far as I can tell, the underlying issue is this: https://bugs.python.org/issue21591 and in my tests it was easily fixed by by replacing the exec() in https://github.com/sqlalchemy/sqlalchemy/blob/master/lib/sqlalchemy/sql/lambdas.py#L1067 with six.exec_(). (I.e. the root of the problem is that on this python2 version there is a difference between the function and the statement version of exec.

I appreciate that python2 support is on the verge of being removed, but then again this is a simple fix and I would much rather migrate to sqlalchemy 1.4 first with py2 while I'm tackling the changes for python3

@zzzeek
Copy link
Member

zzzeek commented Mar 22, 2021

if it's not hard to fix then someone can please send us a PR and we can have that right in for 1.4.3. thanks!

dwt added a commit to dwt/sqlalchemy that referenced this issue Mar 22, 2021
dwt added a commit to dwt/sqlalchemy that referenced this issue Mar 22, 2021
@dwt
Copy link
Contributor

dwt commented Mar 22, 2021

@zzzeek See #6112 for a pull request. I'm not sure though how to provide a meaningful test for this, without having a test executor that also runs python 2.7.5. Please advise what you need there.

@sqla-tester
Copy link
Collaborator

Martin Häcker has proposed a fix for this issue in the master branch:

Fixes #6069 https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/2664

@zzzeek
Copy link
Member

zzzeek commented Mar 22, 2021

wow great catch, we already had exec_() to do this, very nice. I am not too concerned about testing if folks with old Python 2.7's can confirm it works that's good enough for me.

@zzzeek
Copy link
Member

zzzeek commented Mar 22, 2021

when I first saw this error I was concerned it was more about the specific manipulations I'm doing with the code objects.

@dwt
Copy link
Contributor

dwt commented Mar 23, 2021

No, AFAIK it's really all about statement based exec vs function exec(). It works for me - so…

A test run on my build server with this change using tox -e py27-sqlite reports these errors:

FAILED test/sql/test_from_linter.py::TestLinter::test_warn_anon_cte - OperationalError: (sqlite3.OperationalError) near "WITH": syntax error
FAILED test/orm/test_query.py::SessionBindTest::test_sql_expr_bundle_cte_from_entity - OperationalError: (sqlite3.OperationalError) near "WITH": syntax error
FAILED test/orm/test_query.py::SessionBindTest::test_sql_expr_cte_from_entity - OperationalError: (sqlite3.OperationalError) near "WITH": syntax error
FAILED test/orm/test_utils.py::AliasedClassTest::test_meta_getattr_two - RuntimeError: dictionary changed size during iteration
FAILED test/ext/test_hybrid.py::PropertyOverrideTest::test_override_setter - RuntimeError: dictionary changed size during iteration
FAILED test/orm/test_options.py::OfTypePathingTest::test_oftype_only_col_attr_unbound - RuntimeError: dictionary changed size during iteration
FAILED test/orm/test_options.py::OfTypePathingTest::test_oftype_only_rel_attr_string_bound - RuntimeError: dictionary changed size during iteration
FAILED test/orm/test_options.py::OptionsTest::test_from_base_to_subclass_attr - RuntimeError: dictionary changed size during iteration
FAILED test/orm/test_options.py::OptionsTest::test_of_type_plus_level - RuntimeError: dictionary changed size during iteration

Not quite sure what to make of these, as they seem unrelated? Locally with python 2.7.16 these tests pass (with the same change).

@dwt
Copy link
Contributor

dwt commented Mar 23, 2021

What kind of test run do you require to check that these changes are working?

@zzzeek
Copy link
Member

zzzeek commented Mar 23, 2021

a few of those errors look like a very old bundled version of sqlite which would be expected on a very old Python version. the "dictionary sized" errors look much more concerning and are likely more bugs in some combination of SQLAlhcemy and/or very old Python versions but are likely not related to this exec change.

the change passed on CI which includes Python 2.7 so this should get a changelog file and be merged for the next release.

@dwt
Copy link
Contributor

dwt commented Mar 23, 2021

@zzzeek like this? 4e951f9

@sqla-tester
Copy link
Collaborator

Martin Häcker has proposed a fix for this issue in the master branch:

Use compat.exec_() https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/2664

@zzzeek
Copy link
Member

zzzeek commented Mar 23, 2021

glad we could figure this out thanks all for the help!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
awaiting info waiting for the submitter to give more information bug Something isn't working regression something worked and was broken by a change sql
Projects
None yet
Development

No branches or pull requests

6 participants