[BUG] Failing on Python 2.7 + Travis CI with unicode error

[LevN0]

I ran tests a few days ago, prior to release of Tox 3.8, that were passing. Running test suite on basically identical software started failing with a Unicode Error in Tox,

https://travis-ci.org/LevN0/pds4_tools/jobs/513303961

File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/tox/action.py", line 107, in popen
lines = out_path.read().split("\n")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7297: ordinal not in range(128)

Possibly connected to having unicode characters in a filename with-in the package.

Only fails on Python 2.7, succeeds on all Python 3 builds.

[sampsyo]

We're running into this too: https://travis-ci.org/beetbox/beets/jobs/513379693

The problem occurs with Tox 3.8.2 and 3.8.3 but not 3.8.1. Specifically, I believe this change introduced the bug: b5105b1#diff-57c0f90ac19d5a79d6495ab22fab6539R107

The code used to just do this:

output = out_path.read()

But now it does this:

lines = out_path.read().split("\n")
output = "\n".join(lines[3:])

Because that file has unicode_literals enabled, those "\n" string literals are Unicode strings. But because we're on Python 2, the result of read()ing a file is a bytestring.

I think the right fix is to use bytestrings for those "\n" literals on Python 2 but Unicode strings on Python 3. This is sometimes called "native strings," and unicode_literals makes this hard. Future has as_native_str, but I don't (on a cursory glance) see an equivalent utility in six.

[asottile]

@sampsyo no need for a special library, to get a native-literal just use str("\n"). however I took a different approach in #1237 (use the 3.x behaviour in python2 as well)

[sampsyo]

Excellent point, and yes, your fix is better. 👍