Use os.device_encoding(0) to figure out stdin encoding in run_command() (Fix #1777)

[devdanzin]

The output of subprocess.Popen on Windows might include bytes that cannot be decoded by UTF-8.

This PR uses os.device_encoding(0) to figure out the real encoding of stdin (fd 0) so run_command can use that to decode the output. Since os.device_encoding may return None, we fallback to locale.getpreferredencoding.

Fixes #1777.

[nedbat]

It wasn't until you mentioned stdin here that I remembered that 0 is stdin. Wouldn't it be better to check stdout (1)? I doubt they'd be different, but we will be processing the output of a command.

[devdanzin]

Wouldn't it be better to check stdout (1)? I doubt they'd be different, but we will be processing the output of a command.

Indeed, it makes sense that it would be better to check fd 1. I tried a little experiment and changing the encoding of stdout in a terminal makes a difference, but that of stdin doesn't. Here's some notes for (my own) future reference.

Here's the test script I used:

import os, subprocess

proc = subprocess.Popen('echo é𐍈', shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
out, _ = proc.communicate()
stdin_enc, stdout_enc = os.device_encoding(0), os.device_encoding(1)

print(out)
print(stdin_enc, stdout_enc)
# print(out.decode("utf-8"))
print(out.decode(stdin_enc))
print(out.decode(stdout_enc))

Running it in a default terminal will yield:

b'\x82??\r\n'
cp850 cp850
é??

é??

To only change the stdout encoding in PowerShell, run:

[Console]::OutputEncoding = New-Object System.Text.UTF8Encoding

Running it in a terminal with utf-8 stdout results in:

b'\xc3\xa9\xf0\x90\x8d\x88\r\n'
cp850 cp65001
├®­Éìê

é𐍈

I think this means the important encoding is that of stdout.

[nedbat]

Sorry to drag this out, but output_encoding() is only used in this function, so let's move the improved encoding choice code into output_encoding. That way we're decoding with the same encoding we've set in the environment.

[nedbat]

I've pushed a commit to remove the helper and use the same encoding in both places.