Support for Windows 10 built-in ANSI cmd

[Slayther]

Windows 10 has built-in support for ANSI cmd. The ANSI codes should not be stripped.
The module should check for this. I know how to, but I don't know where to put this check..

[wiggin15]

Helo @Slayther. Is colorama broken on Windows 10 cmd, or do both coloring methods (ANSI codes and API calls) work?

The decision about whether to strip and/or convert ANSI codes is in ansitowin32.py (currently lines 74 and 79). It would be great if you could submit a pull request. Otherwise please specify how to check for Windows 10 ANSI code support here. Thanks!

[Slayther]

Well, both ANSI codes and API calls work on new Windows 10 cmd. I thought I saw the dim option working on the new cmd, but I was mistaken once I commented out both ANSI stripping and API calls. Everything else works as expected on Windows.

Nevertheless, if Windows updates support for ANSI sequences, it would be good to support those right away.

I will submit the pull request shortly, once I test some stuff out.

[Slayther]

Fixed with #105.

[vidartf]

While simple test runs of colorama works well on the new Windows 10 cmd, more complex usage of colorama breaks. E.g. calling py.test works well, calling py.test with a specific file gives broken colors, sometimes even with black text on black background (also, Ctrl+C leaves console in whatever color state it was in when exception occurred). Other applications which use colorama also act strange. I unfortunately don't know enough about how colorama works to create a minimal working example.

But yeah, +1 from me on getting a fix in :)

[vidartf]

As colorama is broken in many cases for me on Windows 10, and the submitted PR is still open, I would propose to reopen this issue.

[Slayther]

I did not check the PR as this was a temporary fix before I started using the key codes themselves. @wiggin15 As the build is passing and the feature is proven to work, could you pull the request?

[wiggin15]

@vidartf it would be very helpful if you can produce minimal test cases anyway so we can find exactly what is broken. The behavior on Windows 10 should not be different than the behavior on other Windows versions if using the API for color conversion. I think if we can leave things the way they are without adding detection for Windows 10 it would be better.

[vidartf]

Lifting the relevant bits from py.test, I was able to come up with a minimal example:

import sys
import os
from tempfile import TemporaryFile


# Capture stdout using file descriptors
stdoutfd = 1
saved_fd = os.dup(stdoutfd)
f = TemporaryFile()
os.dup2(f.fileno(), stdoutfd)

# Import colorama with captured stdout
import colorama

# Restore stdout
os.dup2(saved_fd, stdoutfd)
os.close(saved_fd)
f.close()

# Now try to use colorama as ususal:
fil = colorama.AnsiToWin32(sys.stdout).stream

# Normal output
fil.write('Some test output here\n')
# Output gray instead of bold, and on background set to black
fil.write(colorama.Style.BRIGHT + 'Some test output here' + colorama.Style.RESET_ALL + '\n')
# Output black on black
fil.write('Some test output here\n')

Screenshot (from PS, but normal cmd behaves the same):

As you can see, the subsequent prompt is also left with the same colors (black on black). This can be restored by going into properties menu (colors tab).

[wiggin15]

Hi @vidartf thank you for providing this example.
A few comments:

• fil.write('Some test output here\n') (the last line): this looks like it should output normal output, not black on black. Are you missing the ANSI sequences here? When I run your example, it outputs regular output, then bold output, and then regular output again
• The fact that the "black on black" code (if it is there) is not reset and continues to apply on the prompt after the program exits is the expected behavior. If you don't use the "reset" sequence after printing the output, then the terminal continues to use the colors you specified. This is the behavior on unix terminals too. You must reset the colors using colorama.Fore.RESET/colorama.Back.RESET/colorama.Style.RESET_ALL after you are done with the colors. If you use colorama.init() in the beginning of your program then colorama auto-resets everything when the program terminates.
• I understand that this code comes from py.test but they should use the ANSI codes correctly, including the reset. I'm not sure why they're using AnsiToWin32 and printing the output codes directly. colorama provides a simpler interface. The example can be rewritten like this:

import colorama
colorama.init()
print("Some test output here")
print(colorama.Style.BRIGHT + "Some test output here" + colorama.Style.RESET_ALL)
print("Some test output here")

• In any case, I don't think there is anything in this example that is specific to Windows 10. Did you try to run this on an older Windows version?

[vidartf]

Answer point by point:

• The last line should output normally, but outputs black on black. No idea why, but it behaves as expected if colorama is imported when stdout is not redirected.
• I get that a missing reset causes the colors to stay (and agree that the auto-reset functionality is handy). However, in the example, the reset code is included, but doesn't reset correctly. I've updated the example above to use the enums (same behavior).
• Only the point of capturing stdout via file descriptors come from py.test (and the AnsiToWin32 use). The rest is a minimal representation of what happens (see https://github.com/pytest-dev/pytest/blob/master/_pytest/capture.py#L335-L340 for a good place to start understanding corresponding code in pytest).
• On windows 7, the above code prints (without colors):

Some test output here
←[1mSome test output here←[0m
Some test output here

Adding colorama.init() directly after the import prints this (again, without color):

Some test output here
Some test output here
Some test output here
←[0m

Moving the import fixes this (gives colors as expected). As such, I think you might be right that this is not a Win10 issue, but rather that the consequences on win 10 are different from other windows versions.

As extra information:

• py.test normally doesn't have this issue, it only appears if the code to be tested needs to import colorama in a pre-testing step (otherwise py.test imports colorama when the output is not redirected).

[wiggin15]

@vidartf thanks for the explanation. I understand. Further testing is needed to determine exactly what is wrong with the conversion on Windows 10.

[vidartf]

Note: If importing colorama when stdout is captured by FDs, and init() is called, colors will be set to black on black at program end, even without printing anything.

In other words, the following code will set terminal colors to black on black:

import sys
import os
from tempfile import TemporaryFile


# Capture stdout using file descriptors
stdoutfd = 1
saved_fd = os.dup(stdoutfd)
f = TemporaryFile()
os.dup2(f.fileno(), stdoutfd)

# Import colorama with captured stdout
import colorama
colorama.init()

# Restore stdout
os.dup2(saved_fd, stdoutfd)
os.close(saved_fd)
f.close()

[wiggin15]

The issue lies in WinTerm (initialized on import) which sets the default console attributes on its initialization:

self._default = win32.GetConsoleScreenBufferInfo(win32.STDOUT).wAttributes

This default value gets set to 0 when stdoutfd is set to the TemporaryFile's fd. This later causes the RESET_ALL in the example, and at the end of the program (init installs an atexit hook to call reset all) to set the console's properties "back" to 0 (black on black).
I'm not sure why this happens on Windows 10 only.
I'm also not sure why py.test use this redirection method with switching the file descriptors around - it makes sense that the default will break like this. We'll need to find a way to work around this.

[vidartf]

Not sure if this only happens on Windows 10 (that's the only one I got available). Py.test only uses this type of redirection at limited points during import (or when explicitly asked by the test code). Do you see a reasonable workaround for this?

[dragon512]

I was going to add an issue, but maybe should just add my thoughts on this issue.

We need init() to have an option to not use the class wrapper, but behave the same as if it was on Linux. GIven the modern terminal setup on windows supports this by default, this is a better way to go forward and will allow certain combinations to work as strip=True on the init function to work so I can control if I want to suppress color output ( which is current broken on windows) when using colorama

[tartley]

This issue has hopefully been addressed in a way that keeps everyone happy by the recent merging of a new alternative to 'init', called 'just_fix_windows_console', which is now described in the README as the preferred way to use Colorama for most users. This retains the existing complex idiocyncratic behavior of 'init' for those who rely on it being the way it is.