Mock the new module API on <38 and ignore second argument. #11593

Carreau · 2019-02-05T18:21:00Z

As ast.Module is used only in interactiveshell the mock is only a small
lambda function but would need to grow into a proper shim if used more
widely.

Carreau · 2019-02-05T18:21:40Z

@njsmith mind giving it a try ? I'm going to guess ast.changes will will likely have ripple in Jedi as well.

tacaswell · 2019-02-05T18:24:40Z

What behavior should this effect?

Carreau · 2019-02-05T18:28:07Z

None, this should just fix things on 3.8

As ast.Module is used only in interactiveshell the mock is only a small lambda function but would need to grow into a proper shim if used more widely.

tacaswell · 2019-02-05T18:29:35Z

I also run 3.8 on my home machine 😈

Carreau · 2019-02-05T18:30:34Z

damn, via conda ? I some point I had a conda recipe that was faking a master as being a 3.6... but it was breaking too often.

tacaswell · 2019-02-05T18:39:38Z

Via a terrible shell script (https://gist.github.com/tacaswell/a701178de7fcc0c213c0adb0abd57dff ). Basically build cpython, create a venv, pip install into it.

Ah, I now see the issue, kinda hard to miss.... 🐑

tacaswell · 2019-02-05T18:41:03Z

And yes, this branch fixes it.

njsmith · 2019-02-05T22:03:04Z

Change looks good to me. Should you be adding 3.8-dev to your Travis matrix to catch things like this in the future?

Carreau · 2019-02-06T16:11:43Z

I've added 38-dev, but as we are failing on nightly with the ast transformers, i'm going to assume it will fail as well, and someone need to investigate why some transformations are not being applied.

Carreau · 2019-02-07T02:53:10Z

and so apparently Num has been renamed to Constant in the ast ?

Carreau · 2019-02-07T03:02:38Z

Ok, 3 more test fixed. @takluyver, I might need help on some of these errors.

Carreau · 2019-02-07T03:11:45Z

It's unclear to me what's the difference between node.n and node.value are they alias to each other now ? If so should one be deprecated ?

It also seem that both Num, and Str are now folded into the Constant ast node.

Carreau · 2019-02-07T03:13:57Z

@asmeurer , @williamstein don't you have ast transformers ? If so that will affect you.

…ant ast node Also fix soem new sytax warnings.

Carreau · 2019-02-07T16:41:38Z

Ok, test are passing. I'd love for someone to make sure the code make sens, I wrote that by chunks of 5 min here and there so it might not be super coherent.

asmeurer · 2019-02-07T18:11:57Z

I believe we are still using custom transformers based on tokenize. https://github.com/sympy/sympy/blob/master/sympy/interactive/session.py

njsmith · 2019-02-07T21:54:02Z

IPython/core/tests/test_interactiveshell.py

    def visit_Num(self, node):
        node.n = -node.n
        return node

+    if sys.version_info > (3,8):


I don't think this if is required (and likewise for several others below). I'd probably just put comments # for python 3.7 and earlier, # for python 3.8+

Thanks for the review, I added this to every relevant point.

njsmith · 2019-02-07T21:55:24Z

IPython/core/tests/test_ultratb.py


 class TokenizeFailureTest(unittest.TestCase):
    """Tests related to https://github.com/ipython/ipython/issues/6864."""

+    @skipif(sys.version_info > (3,8))


I can't review this line because I have no idea why it's here :-)

This was apparently testing we properly handle a bug in tokenizer that was fixed in 3.8, so the test is not relevant anymore.

Carreau · 2019-05-31T02:08:01Z

Hi there, You seem to run IPython on Python 3.8; do you mind trying #11713 that tried to make use of the new 3.8b1 (not out yet) top-level await ?

### What is this PR for? The root cause is due to ast api is changed after python 3.8, see ipython/ipython#11593. * The PR fixed this issue in `zeppelin_python.py`. * This PR add tests for Python 3.8 for both python interpreter & spark interpreter (spark 2.4 doesn't support python 3.8, so there's no python 3.8 test for spark 2.4). `jupyter_client` 5 is used because jupyter_client 7 doesn't work (tracked in ZEPPELIN-5533) * bokeh test is disabled for flink interpreter. Because pyflink's dependencies conflicts with bokeh2. ### What type of PR is it? [Bug Fix ] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-5525 ### How should this be tested? * Ci pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <zjffdu@apache.org> Closes #4230 from zjffdu/ZEPPELIN-5525 and squashes the following commits: b816a53 [Jeff Zhang] [ZEPPELIN-5525] Python vanillar interpreter doesn't' work in Python 3.8

### What is this PR for? The root cause is due to ast api is changed after python 3.8, see ipython/ipython#11593. * The PR fixed this issue in `zeppelin_python.py`. * This PR add tests for Python 3.8 for both python interpreter & spark interpreter (spark 2.4 doesn't support python 3.8, so there's no python 3.8 test for spark 2.4). `jupyter_client` 5 is used because jupyter_client 7 doesn't work (tracked in ZEPPELIN-5533) * bokeh test is disabled for flink interpreter. Because pyflink's dependencies conflicts with bokeh2. ### What type of PR is it? [Bug Fix ] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-5525 ### How should this be tested? * Ci pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <zjffdu@apache.org> Closes #4230 from zjffdu/ZEPPELIN-5525 and squashes the following commits: b816a53 [Jeff Zhang] [ZEPPELIN-5525] Python vanillar interpreter doesn't' work in Python 3.8 (cherry picked from commit fa1f728) Signed-off-by: Jeff Zhang <zjffdu@apache.org>

### What is this PR for? The root cause is due to ast api is changed after python 3.8, see ipython/ipython#11593. * The PR fixed this issue in `zeppelin_python.py`. * This PR add tests for Python 3.8 for both python interpreter & spark interpreter (spark 2.4 doesn't support python 3.8, so there's no python 3.8 test for spark 2.4). `jupyter_client` 5 is used because jupyter_client 7 doesn't work (tracked in ZEPPELIN-5533) * bokeh test is disabled for flink interpreter. Because pyflink's dependencies conflicts with bokeh2. ### What type of PR is it? [Bug Fix ] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-5525 ### How should this be tested? * Ci pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <zjffdu@apache.org> Closes apache#4230 from zjffdu/ZEPPELIN-5525 and squashes the following commits: b816a53 [Jeff Zhang] [ZEPPELIN-5525] Python vanillar interpreter doesn't' work in Python 3.8

### _Why are the changes needed?_ to close #3795 Python ast api is changed after python 3.8, see ipython/ipython#11593 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #3797 from cfmcgrady/kyuubi-3795. Closes #3795 44edcea [Fu Chen] Revert "debug python version" f8171b0 [Fu Chen] Revert "print ga debug info" 16fde4e [Fu Chen] debug python version 331602a [Fu Chen] print ga debug info 66eeb3f [Fu Chen] python 3.8+ support Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org>

Carreau added this to the 7.3 milestone Feb 5, 2019

tacaswell mentioned this pull request Feb 5, 2019

gallery generation broken on cpython master sphinx-gallery/sphinx-gallery#442

Closed

Mock the new module API on <38 and ignore second argument.

27bd654

As ast.Module is used only in interactiveshell the mock is only a small lambda function but would need to grow into a proper shim if used more widely.

Carreau force-pushed the ast-38 branch from 0352335 to 27bd654 Compare February 5, 2019 18:28

Carreau mentioned this pull request Feb 5, 2019

Mark failing IPython multierror tests as xfail on 3.8 python-trio/trio#905

Merged

test on 3.8-dev

bf6fe70

Num ast has been renamed to Constant on 3.8

954f683

Attemt to fix ast transformer now that Num and Str atr the same Const…

a0dffa8

…ant ast node Also fix soem new sytax warnings.

Carreau force-pushed the ast-38 branch from 4fbeca7 to a0dffa8 Compare February 7, 2019 03:26

Carreau added 2 commits February 6, 2019 19:50

Return current node instead of None

3413d5d

Skip tests that now seem to pass on 3.8.

b8ea86b

Carreau requested a review from takluyver February 7, 2019 16:40

njsmith reviewed Feb 7, 2019

View reviewed changes

comment why the changes and for which python version

357c013

Carreau merged commit 542cc5a into ipython:master Feb 18, 2019

Carreau mentioned this pull request Feb 18, 2019

IPython broken on 3.8-dev, due to changes to ast.Module #11590

Closed

Carreau deleted the ast-38 branch May 31, 2019 02:08

gabrielmagno mentioned this pull request Jan 6, 2021

[LIVY-795] Add support for PySpark with Python >= 3.8 apache/incubator-livy#314

Merged

zjffdu mentioned this pull request Sep 22, 2021

[ZEPPELIN-5525] Python vanillar interpreter doesn't' work in Python 3.8 apache/zeppelin#4230

Closed

1 task

cfmcgrady mentioned this pull request Nov 10, 2022

[KYUUBI #3795][PYSPARK] Python 3.8+ support apache/kyuubi#3797

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mock the new module API on <38 and ignore second argument. #11593

Mock the new module API on <38 and ignore second argument. #11593

Carreau commented Feb 5, 2019

Carreau commented Feb 5, 2019

tacaswell commented Feb 5, 2019

Carreau commented Feb 5, 2019

tacaswell commented Feb 5, 2019

Carreau commented Feb 5, 2019

tacaswell commented Feb 5, 2019

tacaswell commented Feb 5, 2019

njsmith commented Feb 5, 2019

Carreau commented Feb 6, 2019

Carreau commented Feb 7, 2019

Carreau commented Feb 7, 2019

Carreau commented Feb 7, 2019

Carreau commented Feb 7, 2019

Carreau commented Feb 7, 2019

asmeurer commented Feb 7, 2019

njsmith Feb 7, 2019

Carreau Feb 18, 2019

njsmith Feb 7, 2019

Carreau Feb 18, 2019

Carreau commented May 31, 2019

Mock the new module API on <38 and ignore second argument. #11593

Mock the new module API on <38 and ignore second argument. #11593

Conversation

Carreau commented Feb 5, 2019

Carreau commented Feb 5, 2019

tacaswell commented Feb 5, 2019

Carreau commented Feb 5, 2019

tacaswell commented Feb 5, 2019

Carreau commented Feb 5, 2019

tacaswell commented Feb 5, 2019

tacaswell commented Feb 5, 2019

njsmith commented Feb 5, 2019

Carreau commented Feb 6, 2019

Carreau commented Feb 7, 2019

Carreau commented Feb 7, 2019

Carreau commented Feb 7, 2019

Carreau commented Feb 7, 2019

Carreau commented Feb 7, 2019

asmeurer commented Feb 7, 2019

njsmith Feb 7, 2019

Choose a reason for hiding this comment

Carreau Feb 18, 2019

Choose a reason for hiding this comment

njsmith Feb 7, 2019

Choose a reason for hiding this comment

Carreau Feb 18, 2019

Choose a reason for hiding this comment

Carreau commented May 31, 2019