Modernize Python 2 code to get ready for Python 3 #1571

eirnym · 2018-10-14T19:25:11Z

C

Checklist

npm install && npm test passes
tests are included
documentation is changed or added
commit message follows commit guidelines

Description of change

Continuation of #1335 and #1150. Please, review, comment, add more changes and fixes :) All changes are splitted, as some of them are massive.

UPDATE 1
Now I'm fighting with test/test-find-python.js file to allow it even to start testing in Python3-only environment.

UPDATE 2
Tests are passing, but it's possible that I'd miss something

cclauss

Great stuff... Just two minor change requests. Thanks.

cclauss · 2018-10-14T19:51:51Z

gyp/pylib/gyp/simple_copy.py

-for x in (type(None), int, long, float,
-          bool, str, unicode, type):
+import sys
+if sys.version_info < (3, 0):


On this one, can we please follow Python porting best practice use feature detection instead of version detection?

extracted checks to compat module

cclauss · 2018-10-14T19:53:03Z

gyp/pylib/gyp/xcodeproj_file.py

@@ -155,6 +155,9 @@
  _new_sha1 = sha.new


+if sys.version_info >= (3, 0):  # just easier
+    unicode = str


On this one, can we please follow Python porting best practice use feature detection instead of version detection?

There's a reason for it in this case: exception -> substitution will run much slower than knowlege of version where this name doesn't exist. This is also used in six and works well.

so I'd prefer to leave this kind of check for types and for a dictionary. But I can change if you'd insist on it.

refack · 2018-10-14T20:49:22Z

I'm going to land @rodrigc's #1150, then I'll rebase this to see what's left.

eirnym · 2018-10-14T20:57:00Z

@refack it won't work as I found a lot more issues in code to be ported

eirnym · 2018-10-14T20:57:53Z

This doesn't mean, his work is useless.

eirnym · 2018-10-14T21:00:06Z

@refack @cclauss Could you have a chance to know what these variables should be? I haven't found any traces of them in functions they are and how to fix these lines.

./gyp/pylib/gyp/xcode_emulation.py:846:9: F821 undefined name 'cflags'
        cflags.append('-F' + platform_root + '/Developer/Library/Frameworks/')
        ^
./gyp/pylib/gyp/generator/make.py:1645:50: F821 undefined name 'target'
      print("WARNING: no output for", self.type, target)
                                                 ^
2     F821 undefined name 'target'
2

cclauss · 2018-10-14T21:09:22Z

See #1333 and #1334 My sense on cflags is that it should be ldflags in this context but not 100% certain.

refack · 2018-10-14T21:11:30Z

it won't work as I found a lot more issues in code to be ported

As long as it passes CI, I take it as a sign it doesn't regress. I assumed that work was not done yet to get actual py3 compat.
Let's see is we can get everything done in this PR.

eirnym · 2018-10-14T21:15:13Z

My PR also passes CI, but not finished :D There's many things to fix

Before you try to merge it, I'd like to finish some other checks. I'll let you know

eirnym · 2018-10-14T21:32:31Z

Merged and I've done for today.

@cclauss #1333 is included

Unresolved questions for the next run are:

sorted() and cmp=
input.py: compile.ast -> ast. It's about to rewrite main part of the module.
optparse may be dropped in a foreseeable future
mac_tool uses veeery old version of plistlib and implies that pyobjc is installed (which may be not if we install python from MacPorts/Homebrew)
I crashed pylint

eirnym · 2018-10-14T21:46:21Z

@refack I finished, including merging

eirnym · 2018-10-14T21:47:35Z

@cclauss for this run I used pylint --disable=E1125,E0702,E0702,E1701,E1124,C,W0311 pylib after flake8

refack · 2018-10-14T21:55:46Z

@eirnym I've rebased you branch. Before you continue, take a tag, and pull with --rebase, so you can compare what changed.

eirnym · 2018-10-14T22:16:05Z

you lost at least a half of my changes during rebasing. Merge would be always easier.

refack · 2018-10-14T22:22:52Z

you lost at least a half of my changes during rebasing. Merge would be always easier.

We keep a linier git history, so we rebase. Some changes were incompatible with #1572 so I did not base them.

eirnym · 2018-10-14T22:25:11Z

If you'd tell me before, It would be just easier to rebase them manually. Now I repeat my work.

refack · 2018-10-14T22:29:53Z

you lost at least a half of my changes during rebasing. Merge would be always easier.

Sorry. I'm juggling several branches, and reviewing a ton of code...

eirnym · 2018-10-14T22:34:58Z

I understand, but tell about this to a developer to save their time. It's already half past midnight and I'm restoring changes for an hour by now instead going to sleep.

refack · 2018-10-14T22:37:54Z

I understand, but tell about this to a developer to save their time. It's already half past midnight and I'm restoring changes for an hour by now instead going to sleep.

Sorry again.
There is no rush... this could wait a few more days.

Thank you for the dedication.

eirnym · 2018-10-14T22:38:45Z

Next possible day will be in a few days and I don't want to forget and start all things over again.

eirnym · 2018-10-14T22:57:11Z

I restored changes. Commit is a bit bulky, but it does it's job right.

refack · 2018-10-14T23:08:33Z

Thank you very much 🥇

eirnym · 2018-10-18T18:58:25Z

So what the current state of the PR?

eirnym · 2018-10-20T23:58:44Z

@refack @cclauss

cclauss · 2018-10-21T06:50:15Z

Conflicts.

I have approved but I am not a maintainer. Overall, I would feel more comfortable if we used the six module (or futurize) instead of inventing our own from scratch. The linter directives in gyp/pylib/gyp/compat.py are just as a result of a refusal to follow Python porting best practice use feature detection instead of version detection. The map()s and filter()s would be faster and more readable as list comprehensions.

eirnym · 2018-10-21T07:57:57Z

@cclauss I feel the same about six, but I'm not a maintainer to make a decision about additional dependency

Checking Python version is significantly faster and stable enough and it's used even in pip and six.

about map and filter. I changed as little as it was possible. "this is not making code clean and fast, but only a compatibility in mind". if you start make this optimisations, you'll end up with rewriting the whole code from scratch leaving only required features. This is a huge work and I'm not a person to take it in 100%.

eirnym · 2018-10-21T07:59:55Z

And please, remember that we have no tests. npm run test is not sufficient here.

eirnym · 2018-10-21T08:13:36Z

One more thing about compatibility and Python version checking: this is only about syntax and builtins checking. For Python2 you can't use async/await as well as for Python 3 you can't use print statement and very old variant of raising. With imports it's try-except impoting which is basically feautre-checking and never monkey-patching.

eirnym · 2018-10-25T11:30:07Z

@refack almost 2 weeks passed since I done this work. Please, tell me what else needs to be done here?

cclauss · 2018-11-15T16:59:59Z

Conflicts :-(

eirnym · 2018-11-18T13:41:03Z

@cclauss it's easy to solve code conflicts, but I don't see any intention from maintainers to merge these changes anyway.

eirnym · 2018-11-18T13:49:06Z

I gave up on this branch. I have a copy if anybody would be intereseted in merging it.

eirnym mentioned this pull request Oct 14, 2018

Modernize Python 2 code to get ready for Python 3 #1335

Merged

2 tasks

cclauss requested changes Oct 14, 2018

View reviewed changes

eirnym mentioned this pull request Oct 14, 2018

Add some Python 3 fixes to node-gyp #1150

Merged

refack self-assigned this Oct 14, 2018

refack force-pushed the master branch from 63e6bc7 to ef8b60a Compare October 14, 2018 21:19

cclauss approved these changes Oct 15, 2018

View reviewed changes

gyp: more py3 compat

d713478

blink1073 mentioned this pull request Nov 7, 2018

info fsevents@1.1.3: The platform "win32" is incompatible with this module. jupyterlab/jupyterlab#3462

Closed

eirnym closed this Nov 18, 2018

Modernize Python 2 code to get ready for Python 3 #1571

Modernize Python 2 code to get ready for Python 3 #1571

Conversation

eirnym commented Oct 14, 2018 • edited by refack

Checklist

Description of change

cclauss left a comment

Choose a reason for hiding this comment

cclauss Oct 14, 2018

Choose a reason for hiding this comment

eirnym Oct 14, 2018

Choose a reason for hiding this comment

cclauss Oct 14, 2018

Choose a reason for hiding this comment

eirnym Oct 14, 2018

Choose a reason for hiding this comment

eirnym Oct 14, 2018 • edited

Choose a reason for hiding this comment

refack commented Oct 14, 2018

eirnym commented Oct 14, 2018

eirnym commented Oct 14, 2018

eirnym commented Oct 14, 2018

cclauss commented Oct 14, 2018

refack commented Oct 14, 2018

eirnym commented Oct 14, 2018

eirnym commented Oct 14, 2018

eirnym commented Oct 14, 2018

eirnym commented Oct 14, 2018

refack commented Oct 14, 2018

eirnym commented Oct 14, 2018

refack commented Oct 14, 2018

eirnym commented Oct 14, 2018

refack commented Oct 14, 2018

eirnym commented Oct 14, 2018

refack commented Oct 14, 2018

eirnym commented Oct 14, 2018

eirnym commented Oct 14, 2018

refack commented Oct 14, 2018

eirnym commented Oct 18, 2018

eirnym commented Oct 20, 2018

cclauss commented Oct 21, 2018

eirnym commented Oct 21, 2018 • edited

eirnym commented Oct 21, 2018

eirnym commented Oct 21, 2018 • edited

eirnym commented Oct 25, 2018

cclauss commented Nov 15, 2018

eirnym commented Nov 18, 2018

eirnym commented Nov 18, 2018

eirnym commented Oct 14, 2018 •

edited by refack

eirnym Oct 14, 2018 •

edited

eirnym commented Oct 21, 2018 •

edited

eirnym commented Oct 21, 2018 •

edited