Fix newlines in config files

[kyluca]

This change should still be sufficient to close #58.

Heavily inspired by the previous patches and PR jtrakk#1, however it appears the original PR wasn't merged in upstream?

[kyluca]

Build failed with a temporary network issue:

ProtocolError: ("Connection broken: error(104, 'Connection reset by peer')", error(104, 'Connection reset by peer'))

Will rebuild

[arielnmz]

Nice, thank you so much

[florisla]

We also have to take care not to change the newline style of the file.

I think that currently this PR will produce a weird result for Linux users who use <carriage return><newline> as line endings in the config file: the very last line ending will miss the carriage return.

Same goes for Windows users who have only <newline> as line ending; the last line will include a carriage return.

See merge request #59 which tries to keep the line endings consistent.

I think it would make sense to add a test like test_retain_newline which specifically tests that the final newline of the file is of the right type.

[kyluca]

Thanks for the suggestion, have updated the tests to explicitly check the final line ending occurs once (not twice) and is of the right type. Also confirmed locally that

• the test failed without the code change (with two line endings at the end of the file)
• and passed with the code change (with one line ending at the end of the file).

The code is currently making use of Python's universal newline conversion handling when using open in text mode.

• During the read operation, we take note of the original line endings and use the default for newline=None so Python converts them to \n for us.
• At the other end during the write operation, we pass the original line endings back with newline=config_newlines to convert \n in the content to \r\n or whatever the original file format was (platform independent).

Excerpt from the docs: https://docs.python.org/3/library/functions.html#open

newline controls how universal newlines mode works (it only applies to text mode). It can be None, '', '\n', '\r', and '\r\n'. It works as follows:

• When reading input from the stream, if newline is None, universal newlines mode is enabled. Lines in the input can end in '\n', '\r', or '\r\n', and these are translated into '\n' before being returned to the caller. If it is '', universal newlines mode is enabled, but line endings are returned to the caller untranslated. If it has any of the other legal values, input lines are only terminated by the given string, and the line ending is returned to the caller untranslated.
• When writing output to the stream, if newline is None, any '\n' characters written are translated to the system default line separator, os.linesep. If newline is '' or '\n', no translation takes place. If newline is any of the other legal values, any '\n' characters written are translated to the given string.

[florisla]

Look good to me!

I'm in favor of merging this.

[florisla]

@ekohl Can you look into this one too?

[avilaton]

Thanks