DRAFT: Apply pyupgrade. #51

Carreau · 2020-08-13T04:45:22Z

Merge Add conversion to f-strings using flynt #308
Rebase after Add conversion to f-strings using flynt #308 has been merged
Evaluate whether the approach in Add conversion to f-strings using flynt #308 fits directly or needs to be adjusted
Unit tests
Documentation

(pyupgrade is "a tool (and pre-commit hook) to automatically upgrade syntax for newer versions of the language.")

Just seeing how it looks like, we would of course need to pass options in, so i'm thinking only once there is a config file; likely.

This is likely incorrect as the edited lines could likely be off once Black has been ran as it can rewrap stuff.

It also does use pyupgrade private internal API, and the author has explicitly stated they they were not willing to have public API beyond the CLI.

We should also likely run this before Black; maybe even before isort? Not sure if it might remove imports... But at least it does not pass the AST unchange part.

Late for today, just submitting in case you decide to also write the same thing.

akaihola · 2020-08-13T06:28:35Z

Oh wow, another splendid idea! But I agree with your notes regarding options, config file first, and the AST check challenge.

Also, I'm not sure if you meant this by "pass options in", but I think this should be off by default and activated using a command line / configuration file option.

akaihola · 2020-08-13T20:59:29Z

src/darker/__main__.py

@@ -125,14 +125,42 @@ def format_edited_parts(
                # 10. A re-formatted Python file which produces an identical AST was
                #     created successfully - write an updated file or print the diff
                #     if there were any changes to the original
+                result_str = pyup(result_str, edited_lines, edited_linenums)


I agree with your note about pyupgrade changes being better done before Black reformatting, maybe even as the first thing before isort.

akaihola · 2020-08-13T21:04:36Z

src/darker/__main__.py

+
+    pyup_opcodes = diff_and_get_opcodes(edited_lines, content)
+    pyupgrade_chunks = list(opcodes_to_chunks(pyup_opcodes, edited_lines, content))
+    pyup_chosen_lines: List[str] = list(choose_lines(pyupgrade_chunks, edited_linenums))


edited_linenums isn't necessarily correct any more at this point. If Black has e.g. divided a long line on multiple lines, every line number after that gets shifted. This would cause wrong pyupgrade changes to be selected and desired ones skipped. This is why changes by all tools (pyupgrade, isort, Black) need to be done first, and only then the diff be calculated between the user's original version and the automatically modified one.

Yes, that what I was thinking, or recomputing the diff WRT the revision after each tool.

I think the problem with (all tools) then (compute diff), is you may end up with chunks larger than you wants.

say original

a = set((1, 2)) b = c

user edit

- b = c + b = d

(black + pyupgrade) then (compute diff), will change both lines, which you don't want to apply.
Though,
(black, diff) should be applied, then (pyupgrade, diff) should not.

There's also the additional potential problem of tools which may change the order of lines in code. isort is an example of such a tool, and that's why Darker just includes modifications from it as part of the starting point, and compares Black results to that. I wonder if any of the rules in pyupgrade could potentially also change the order of lines in the source code?

Here's an illustration of what code reordering would cause if such changes would be attempted to apply to only modified lines:

Let's say I have this module committed to the HEAD of my repository:

1 import os 2 import black 3 import sys

I then manually correct the first line to import os without the extra space:

*1 import os 2 import black 3 import sys

Run isort:

isort myfile.py

Get this as a result:

1 import os *2 import sys *3 *4 import black

Git shows the difference:

git diff -U0

git diff -U0|cat diff --git a/imp.py b/imp.py index 61a152d..1ed0c5e 100644 --- a/imp.py +++ b/imp.py @@ -1,2 +1 @@ -import os -import black +import os @@ -3,0 +3,2 @@ import sys + +import black

Only the first chunk falls on line 1 (the only edited line) and would be applied by a Darker-like tool. The result would be:

1 import os 2 import sys

So I would actually lose my Black import as a result of applying sorted imports partially.

from my look at upgrade that does nto seem to be the case, it really appears to do AST node replacements, so I would not be too worried.

Carreau · 2020-08-13T21:07:59Z

Oh wow, another splendid idea! But I agree with your notes regarding options, config file first, and the AST check challenge.

Thanks !

Also, I'm not sure if you meant this by "pass options in", but I think this should be off by default and activated using a command line / configuration file option.

Yes, off by default would be my preference I think. And the you can opt-in on per-repo config files.
I was also expecting the work on linters you did in the other PR to refactor this area so did not push much on the impl.

akaihola · 2021-05-13T13:39:06Z

@Carreau do you still think this patch would fit Darker's purpose? If so, we probably need to try and rebase on master – I can help there if the conflicts are complicated.

akaihola · 2022-02-25T15:13:12Z

@Carreau, I rebased this on master and removed the actual call to the pyup() function since it needs to be thought through anyway. I only now looked more closely to what you hade already implemented, and I believe we can make this work. Let's see if this happens by version 1.5.0 or only in 1.6.0 of Darker.

akaihola · 2022-08-28T09:17:18Z

Support for flynt is cooking in #308. The same approach might fit well for pyupgrade. Let's rebase this after that one has been merged.

Just seeing how it looks like, we would of course need to pass options in, so i'm thinking only once there is a config file; likely. So far there's just a function to call `pyupgrade` on given source code. Applying it to modified lines needs to be implemented, probably best by refactoring `_reformat_single_file()` so it can be used for different source-modifying tools. It also does use pyupgrade private internal API, and the author has explicitly stated they they were not willing to have public API beyond the CLI. We should also likely run this _before_ black; maybe even before isort ? not sure if it might remove imports... But at least it does not pass the ast unchange part.

akaihola · 2023-01-22T20:45:09Z

Flynt support in #308 has been merged. We can now resume implementing pyupgrade support.

akaihola · 2023-02-11T17:13:33Z

Moving to version 1.8.0. There are a plenty of changes for 1.7.0 already, I now prefer to get it out first.

akaihola added the enhancement New feature or request label Aug 13, 2020

akaihola requested changes Aug 13, 2020

View reviewed changes

Carreau mentioned this pull request Aug 19, 2020

Remove u string prefixes ipython/traitlets#598

Merged

akaihola mentioned this pull request Feb 22, 2022

[DESIGN] support different code formatters, allow to skip formatting #304

Open

akaihola force-pushed the pyupgrade branch from f570629 to 6ca9f10 Compare February 25, 2022 15:11

akaihola force-pushed the pyupgrade branch from 5e47a4d to ae5fb50 Compare February 27, 2022 14:39

akaihola force-pushed the pyupgrade branch 3 times, most recently from 096c525 to e1fcd1f Compare March 14, 2022 17:52

akaihola force-pushed the pyupgrade branch 3 times, most recently from 75685fb to 8d1ceef Compare March 26, 2022 21:42

akaihola force-pushed the pyupgrade branch 5 times, most recently from 0a69c95 to b6d3edd Compare April 5, 2022 16:56

akaihola force-pushed the pyupgrade branch 3 times, most recently from f330bc8 to 720c44e Compare April 10, 2022 12:36

akaihola force-pushed the pyupgrade branch from 720c44e to b26ac8c Compare April 15, 2022 14:47

Repository owner deleted a comment from sourcery-ai bot Apr 24, 2022

akaihola added this to the 1.6.0 milestone Apr 24, 2022

akaihola force-pushed the pyupgrade branch from b26ac8c to a7c7c44 Compare April 24, 2022 19:53

akaihola force-pushed the pyupgrade branch from a7c7c44 to dc5fd71 Compare August 28, 2022 09:23

akaihola force-pushed the pyupgrade branch from 2e51647 to 2d95ffb Compare August 30, 2022 06:21

akaihola modified the milestones: 1.6.0, 1.7.0 Sep 17, 2022

akaihola force-pushed the pyupgrade branch 3 times, most recently from 185047d to fcc55c3 Compare November 15, 2022 06:07

akaihola mentioned this pull request Dec 28, 2022

Add conversion to f-strings using flynt #308

Merged

11 tasks

akaihola force-pushed the pyupgrade branch 3 times, most recently from fcaf05c to 0d5ede9 Compare January 10, 2023 18:33

akaihola force-pushed the pyupgrade branch 3 times, most recently from 5761f7e to bf3656f Compare January 15, 2023 21:17

Carreau and others added 3 commits January 22, 2023 22:43

Add the darker[pyupgrade] extras option

6309929

Move pyupgrade tooling to a dedicated module

2195e75

akaihola force-pushed the pyupgrade branch from bf3656f to 2195e75 Compare January 22, 2023 20:43

akaihola modified the milestones: 1.7.0, 1.8.0 Feb 11, 2023

akaihola modified the milestones: Darker 1.8.0 – Flynt compatibility, Darker 1.9.0 – Graylint Mar 26, 2023

akaihola mentioned this pull request Mar 7, 2024

Support Ruff as a formatter #521

Open

akaihola modified the milestones: Darker 2.2.0 - New feature release, Darker 3.0.0 - Backwards incompatible changes Mar 16, 2024

akaihola assigned Carreau Apr 6, 2024

akaihola modified the milestones: Darker 3.0.0 - Backwards incompatible changes, Darker 2.2.0 - New feature release Apr 6, 2024

akaihola mentioned this pull request Apr 13, 2024

Feature Request: Docker container #580

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DRAFT: Apply pyupgrade. #51

DRAFT: Apply pyupgrade. #51

Carreau commented Aug 13, 2020 •

edited by akaihola

akaihola commented Aug 13, 2020

akaihola Aug 13, 2020

akaihola Aug 13, 2020

Carreau Aug 13, 2020

akaihola Aug 18, 2020

Carreau Aug 19, 2020

Carreau commented Aug 13, 2020

akaihola commented May 13, 2021

akaihola commented Feb 25, 2022

akaihola commented Aug 28, 2022 •

edited

akaihola commented Jan 22, 2023 •

edited

akaihola commented Feb 11, 2023

DRAFT: Apply pyupgrade. #51

Are you sure you want to change the base?

DRAFT: Apply pyupgrade. #51

Conversation

Carreau commented Aug 13, 2020 • edited by akaihola

akaihola commented Aug 13, 2020

akaihola Aug 13, 2020

Choose a reason for hiding this comment

akaihola Aug 13, 2020

Choose a reason for hiding this comment

Carreau Aug 13, 2020

Choose a reason for hiding this comment

akaihola Aug 18, 2020

Choose a reason for hiding this comment

Carreau Aug 19, 2020

Choose a reason for hiding this comment

Carreau commented Aug 13, 2020

akaihola commented May 13, 2021

akaihola commented Feb 25, 2022

akaihola commented Aug 28, 2022 • edited

akaihola commented Jan 22, 2023 • edited

akaihola commented Feb 11, 2023

Carreau commented Aug 13, 2020 •

edited by akaihola

akaihola commented Aug 28, 2022 •

edited

akaihola commented Jan 22, 2023 •

edited