`pip-compile`: Use the same newline string already found in relevant files (if impossible use LF), or override with `--force-lf-newlines` #1584

AndydeCleyre · 2022-02-16T05:51:43Z

This changes pip-compile's default behavior for writing newlines (1), and adds an overriding flag (2).

(1)

By default, pip-compile will determine a newline string to use based on a "preserve" strategy:

if a pre-existing output file has a newline, use that
otherwise, if an input file has a newline, use that
otherwise, use LF (\n)

(2)

pip-compile gains a flag: --force-lf-newlines, which ensures LF is used in the output file.

This aims to address #1448.

Contributor checklist

Provided the tests for the changes.
Assure PR title is short, clear, and good to be included in the user-oriented changelog

Maintainer checklist

Assure one of these labels is present: backwards incompatible, feature, enhancement, deprecation, bug, dependency, docs or skip-changelog as they determine changelog listing.
Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

graingert · 2022-07-18T21:20:07Z

piptools/scripts/compile.py

@@ -159,6 +159,12 @@ def _get_default_option(option_name: str) -> Any:
        "Will be derived from input file otherwise."
    ),
 )
+@click.option(
+    "--force-unix-newlines",


I don't like the name of this flag because it implies that LF is only for Unix, however other operating systems are gradually coming around to LF being the only option (windows)

I'm going to push here an update based on changes in the other PR. What do you think the flag could be called instead?

That's part of the reason I prefer the other PR, the CLI is significantly more understandable at the cost of a tiny number of extra lines, and nobody needs to bikeshed this name

The best I can come up with is:

Suggested change

"--force-unix-newlines",

"--force-lf-line-separator",

Suggested change

"--force-unix-newlines",

"--newline-is-lf",

--newline=lf is so readable and obvious, it just has a drawback that it implies the existence of --newline=crlf

newline-is-lf sounds descriptive, but we're being prescriptive. What about:

write-lf

force-lf

write-lf-newlines

force-lf-newlines

?

graingert · 2022-07-18T21:36:11Z

piptools/scripts/compile.py

@@ -159,6 +159,12 @@ def _get_default_option(option_name: str) -> Any:
        "Will be derived from input file otherwise."
    ),
 )
+@click.option(
+    "--force-unix-newlines",


That's part of the reason I prefer the other PR, the CLI is significantly more understandable at the cost of a tiny number of extra lines, and nobody needs to bikeshed this name

The best I can come up with is:

Suggested change

"--force-unix-newlines",

"--force-lf-line-separator",

Suggested change

"--force-unix-newlines",

"--newline-is-lf",

--newline=lf is so readable and obvious, it just has a drawback that it implies the existence of --newline=crlf

AndydeCleyre · 2022-07-18T21:51:33Z

@ssbarnea I understand you don't really want this in any form, but if it does happen in this (single flag) form, would you like to suggest a better flag name?

EDIT: In this PR it's currently --force-lf-newlines.

Need to use a local repo for the `--force-lf-newlines` option, at least until jazzband/pip-tools#1584 can be merged.

Need to point to a pip-tools fork for the `--force-lf-newlines` option until jazzband/pip-tools#1584 can be merged.

Looks like jazzband/pip-tools#1584 was force-pushed a while back and the reference to the previously used revision hash was lost.

- Use it in writer to override the line sep used in output, using io.TextIOWrapper - Set default to preserve, which checks the output file, then the input file(s) - Falls back to LF if that's not possible

- skip decode to save time - catch FileNotFoundError to avoid potential race condition Co-authored-by: Thomas Grainger <tagrain@gmail.com>

The default behavior remains identical to what was --newline=preserve.

ssbarnea · 2022-10-06T09:15:05Z

I think that this change was obsoleted by #1652 so closing. If not, please rebase and reopen.

AndydeCleyre mentioned this pull request Feb 16, 2022

Add --newline=[LF|CRLF|native|preserve] option to compile, to override the line separator characters used #1491

Closed

4 tasks

AndydeCleyre mentioned this pull request Mar 3, 2022

Add a way to choose line endings in compiled files #1448

Open

AndydeCleyre force-pushed the feature/linesep-slim branch from 24d7e29 to 46a5b15 Compare March 15, 2022 16:11

AndydeCleyre force-pushed the feature/linesep-slim branch 4 times, most recently from 7903d76 to c5470a5 Compare April 4, 2022 20:43

AndydeCleyre force-pushed the feature/linesep-slim branch from c5470a5 to e64b217 Compare April 28, 2022 21:41

AndydeCleyre force-pushed the feature/linesep-slim branch from e64b217 to 91bd957 Compare May 31, 2022 15:23

AndydeCleyre force-pushed the feature/linesep-slim branch from 91bd957 to eb9ade0 Compare June 8, 2022 17:59

AndydeCleyre force-pushed the feature/linesep-slim branch from eb9ade0 to 79af990 Compare June 16, 2022 16:56

AndydeCleyre force-pushed the feature/linesep-slim branch 3 times, most recently from 09344e4 to 1cb6997 Compare July 3, 2022 03:37

AndydeCleyre mentioned this pull request Jul 3, 2022

Add --newline=[LF|CRLF|native|preserve] option to pip-compile #1652

Merged

4 tasks

AndydeCleyre force-pushed the feature/linesep-slim branch from 1cb6997 to efa88f7 Compare July 18, 2022 16:04

This comment was marked as resolved.

Sign in to view

graingert reviewed Jul 18, 2022

View reviewed changes

graingert requested changes Jul 18, 2022

View reviewed changes

AndydeCleyre force-pushed the feature/linesep-slim branch from efa88f7 to 1b4a0d4 Compare July 18, 2022 21:51

AndydeCleyre force-pushed the feature/linesep-slim branch from 1b4a0d4 to ae15c4d Compare July 19, 2022 14:10

AndydeCleyre changed the title ~~Feature/linesep slim~~ pip-compile: Use the same newline string already found in relevant files (if impossible, use LF), or override with --force-lf-newlines Jul 19, 2022

AndydeCleyre marked this pull request as ready for review July 26, 2022 15:32

nuztalgia added a commit to nuztalgia/botstrap that referenced this pull request Aug 9, 2022

Update lock files with local pip-compile pre-commit instead of Renovate.

dfe0fdb

Need to use a local repo for the `--force-lf-newlines` option, at least until jazzband/pip-tools#1584 can be merged.

nuztalgia added a commit to nuztalgia/botstrap that referenced this pull request Aug 9, 2022

Use a pre-commit hook instead of Renovate for pip-compile.

d31be29

Need to point to a pip-tools fork for the `--force-lf-newlines` option until jazzband/pip-tools#1584 can be merged.

AndydeCleyre force-pushed the feature/linesep-slim branch from ae15c4d to bed2edb Compare August 15, 2022 13:46

AndydeCleyre force-pushed the feature/linesep-slim branch 2 times, most recently from 89fa7ee to ded5950 Compare October 3, 2022 15:17

AndydeCleyre requested a review from graingert October 3, 2022 16:16

AndydeCleyre and others added 4 commits October 5, 2022 18:48

Add --newline [LF|CRLF|native|preserve] option to compile

800d481

- Use it in writer to override the line sep used in output, using io.TextIOWrapper - Set default to preserve, which checks the output file, then the input file(s) - Falls back to LF if that's not possible

Add tests for --newline option

71d5248

Move linesep determination to a dedicated function, with improvements

6dad09d

- skip decode to save time - catch FileNotFoundError to avoid potential race condition Co-authored-by: Thomas Grainger <tagrain@gmail.com>

Replace newline choices with --force-lf-newlines flag

d247f43

The default behavior remains identical to what was --newline=preserve.

webknjaz force-pushed the feature/linesep-slim branch from dd85a3b to d247f43 Compare October 5, 2022 16:48

ssbarnea closed this Oct 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`pip-compile`: Use the same newline string already found in relevant files (if impossible use LF), or override with `--force-lf-newlines` #1584

`pip-compile`: Use the same newline string already found in relevant files (if impossible use LF), or override with `--force-lf-newlines` #1584

AndydeCleyre commented Feb 16, 2022 •

edited

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

graingert Jul 18, 2022 •

edited

AndydeCleyre Jul 18, 2022

graingert Jul 18, 2022 •

edited

AndydeCleyre Jul 18, 2022

graingert Jul 18, 2022 •

edited

AndydeCleyre commented Jul 18, 2022 •

edited

ssbarnea commented Oct 6, 2022

pip-compile: Use the same newline string already found in relevant files (if impossible use LF), or override with --force-lf-newlines #1584

pip-compile: Use the same newline string already found in relevant files (if impossible use LF), or override with --force-lf-newlines #1584

Conversation

AndydeCleyre commented Feb 16, 2022 • edited

(1)

(2)

Contributor checklist

Maintainer checklist

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

graingert Jul 18, 2022 • edited

Choose a reason for hiding this comment

AndydeCleyre Jul 18, 2022

Choose a reason for hiding this comment

graingert Jul 18, 2022 • edited

Choose a reason for hiding this comment

AndydeCleyre Jul 18, 2022

Choose a reason for hiding this comment

graingert Jul 18, 2022 • edited

Choose a reason for hiding this comment

AndydeCleyre commented Jul 18, 2022 • edited

ssbarnea commented Oct 6, 2022

`pip-compile`: Use the same newline string already found in relevant files (if impossible use LF), or override with `--force-lf-newlines` #1584

`pip-compile`: Use the same newline string already found in relevant files (if impossible use LF), or override with `--force-lf-newlines` #1584

AndydeCleyre commented Feb 16, 2022 •

edited

graingert Jul 18, 2022 •

edited

graingert Jul 18, 2022 •

edited

graingert Jul 18, 2022 •

edited

AndydeCleyre commented Jul 18, 2022 •

edited