Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Yet another attempt at supporting Windows 10's ANSI/VT console #139

Merged
merged 8 commits into from Oct 16, 2022

Conversation

segevfiner
Copy link
Contributor

This time I'm properly using ENABLE_VIRTUAL_TERMINAL_PROCESSING which is what really controls this. I enable it when first needed and disable it on atexit. Also, instead of checking for Windows versions, I check that enabling ENABLE_VIRTUAL_TERMINAL_PROCESSING works (Probably needs additional testing to make sure this doesn't unexpectedly break).

One obvious benefit is that Windows 10, since the creators update, also implements 256-bit color and 24-color escapes. But there definitely may be many more codes which are supported by Windows but not by colorama.

Note that cmd has some weird behavior with this flag. It enables it when it runs, disables it when it invokes a program, and doesn't disable it when it exits. This might make you think it only works in PowerShell, which simply enables it and leaves it on (This behavior may of course change in future Windows versions).

Should be reviewed

  • I choose to enable this in AnsiToWin32 since I remember some Python libraries that use it directly and completely bypass init (For example py (Used by pytest) and click).
  • I'm resetting _atexit_registered once it's called from atexit, so that in case some code re-enables this during atexit, we will re-register the callback so it will be called again to cleanup.
  • The dodgy conditions for strip/convert, they are so easy to get wrong. Please check and double check me.
  • And, of course, this should really be tested by other Windows users. On both Windows 10 that supports this, and older Windows versions. Maybe while using other libraries that use colorama.

Example

import sys
import colorsys
import colorama
colorama.init()


for i in xrange(256):
    color = colorsys.hsv_to_rgb(i/255., 1, 1)
    color = int(color[0] * 255), int(color[1] * 255), int(color[2] * 255)
    sys.stdout.write('\x1b[48;2;{0};{1};{2}m \x1b[0m'.format(*color))
sys.stdout.write('\n')

image

Previous/Other Attempts

#133
#105

@Redjumpman
Copy link

Redjumpman commented Sep 21, 2017

This is working for me on Windows 10. Python 3.6.2

@Stylite
Copy link

Stylite commented Sep 21, 2017

Works fine for me on Windows 10 and Windows 10 insider build. Python 3.6 and Python 3.6.2

@ddorn
Copy link

ddorn commented Oct 27, 2017

Works fine for me on Windows 10 Pro Fall update with python 3.6.2 ! :D

@Eeems
Copy link

Eeems commented Oct 28, 2017

While this works, both powershell and cmd crash after exiting one of my test programs. I'm utilizing the alternate screen which may be the cause. Try with this test code:

from colorama import init, Fore, Back, Style

init()
print '\x1b[?1049h'  # Alternate screen
print Fore.RED + Style.BRIGHT + '\x1b[4m\x1b[3mSome Text'  # Underline + Bright + Italic
raw_input(Style.RESET_ALL + 'Press Enter')
print '\x1b[?1049l'  # Exit alternate screen

You may have to try to use powershell/cmd after the program exits to see the crash.

I'm running on windows 10 pro version 1703 build 15063.674

Edit: I should note that I'm running this with python 2.7.11

@segevfiner
Copy link
Contributor Author

@Eeems I think you just stumbled upon a bug in Windows 10's VT emulation, this will trigger the crash without colorama (Only Pywin32):

import sys
import win32console


ENABLE_VIRTUAL_TERMINAL_PROCESSING = 0x0004

conout = win32console.GetStdHandle(win32console.STD_OUTPUT_HANDLE)
conout.SetConsoleMode(conout.GetConsoleMode() | ENABLE_VIRTUAL_TERMINAL_PROCESSING)

print '\x1b[?1049h'  # Alternate screen
print '\x1b[31m\x1b[1m\x1b[4m\x1b[3mSome Text'  # Underline + Bright + Italic
raw_input('\x1b[0mPress Enter')
print '\x1b[?1049l'  # Exit alternate screen

I can't seem to find this using Google. Probably want to report this to Microsoft... somewhere...

@Eeems
Copy link

Eeems commented Oct 29, 2017

@segevfiner well great. Who knows how long before that is fixed. I'll poke around as to where I can report it tomorrow and update this with a reference number if I get one.

Best I could find for a place to report these kind of things was the feedback hub. Here is the issue: https://aka.ms/Txnt6p

@tleonhardt
Copy link

I believe this bug you are encountering is the same one referenced here:
microsoft/terminal#162

It looks like a fix is being worked on.

@Eeems
Copy link

Eeems commented Jun 29, 2018

Thanks @tleonhardt

@Eeems
Copy link

Eeems commented Jan 11, 2019

@tleonhardt Looks like it's been shipped with 1809

zmwangx added a commit to zmwangx/googler that referenced this pull request Apr 10, 2019
VT100 control sequences are supported in cmd and PowerShell starting from
Windows 10 Anniversary Update, but only if the
ENABLE_VIRTUAL_TERMINAL_PROCESSING flag is set on the screen buffer handle
using SetConsoleMode.

Setting this flag does not seem to negatively affect third party terminal
emulators with native ANSI support such as ConEmu, so hopefully there's no
regression.

References:
https://docs.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences
https://docs.microsoft.com/en-us/windows/console/setconsolemode

Credits:
jarun#275
tartley/colorama#139
zmwangx added a commit to zmwangx/googler that referenced this pull request Apr 10, 2019
VT100 control sequences are supported in cmd and PowerShell starting from
Windows 10 Anniversary Update, but only if the
ENABLE_VIRTUAL_TERMINAL_PROCESSING flag is set on the screen buffer handle
using SetConsoleMode.

Setting this flag does not seem to negatively affect third party terminal
emulators with native ANSI support such as ConEmu, so hopefully there's no
regression.

References:
https://docs.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences
https://docs.microsoft.com/en-us/windows/console/setconsolemode

Credits:
jarun#275
tartley/colorama#139
@tartley
Copy link
Owner

tartley commented Oct 13, 2020

Hey. FYI, yesterday I created a PR to test releases before we push them to PyPI. When that is merged, I'll be more confident about resuming merges and releases. I'll try to look at this PR soon. Thank you for creating it!

@wagnerpeer
Copy link

Works for me, too.
Windows 10 Enterprise 20H2 running Python 3.9.5

How can I help to get this PR merged?

@tartley
Copy link
Owner

tartley commented Oct 7, 2021

This looks superb, and I'd love to get it merged imminently for a forthcoming release.

Unfortunately right now I can't merge it because it contains a bunch of code changes but no corresponding test changes. How would future contributors find out if their changes break your code?

If nobody else steps up, I'll take a look at fixing the conflicts and adding tests for this change in the next few days. If I don't get to it, then it might not make it into the very next release, but I hope to pick up doing them more regularly going forward...

Apolologies for ignoring this for years. Thank you for your graceful patience.

@tartley tartley added the 0.4.5 Get merged for 0.4.5 release label Oct 7, 2021
@tartley tartley added 0.4.6 Get merged for 0.4.6. release and removed 0.4.5 Get merged for 0.4.5 release labels Jun 14, 2022
@njsmith
Copy link
Collaborator

njsmith commented Aug 24, 2022

@tartley I recently ran into this on a work project where we were setting up the windows console correctly to process our ansi escapes, but then a third-party dependency pulled in colorama and it broke our rendering b/c colorama started intercepting the rendering instead. Whoops :-). Given that colorama is the de facto standard for working around Win32 console weirdness, I think colorama is the right place to fix this. Convincing everyone to stop using colorama would be really hard; but making colorama silently do-the-right-thing on modern Windows seems a lot more doable.

So.... is there any way I can help? I am, sadly, pretty familiar with the annoying details of Win32 (sample code; would you trust me to take over reviewing this, so you don't have to?

For what it's worth:

  • This is the right approach; Provide pass-through ANSI on Windows 10 powershell #133 and Add Win 10 native ANSI cmd support #105 can be closed in favor of it
  • I think the atexit handling here is probably too fragile and nitpicky to be worth it, and we should just unconditionally enable escape processing and leave it enabled. Like, Microsoft's own cmd.exe unconditionally enables ANSI processing when it runs, and doesn't bother restoring the terminal state afterwards. If it's good enough for them, it should be good enough for us :-).
  • Testing is quite tricky, because almost all the actual work is in interfacing with the Win32 console subsystem. So I guess the options are either a fairly-trivial mock test, that doesn't test too much but at least would catch really gratuitous problems? Or else adding wrappers for CreatePseudoConsole and friends to make a real end-to-end test -- but this is pretty complex, because it requires passing a custom structure to CreateProcess etc.

@tartley
Copy link
Owner

tartley commented Aug 25, 2022

@njsmith Hi, I am in awe of your stellar work, many thanks for your attention. What you say makes sense to me.

I'm happy to defer to your judgement about atexit handling. Seems reasonable.

I'm not opposed to the more end-to-end test you describe if you think it's worth it, but if memory serves we use shallow mock/unit tests everywhere else in Colorama, so I'd be happy with that.

In my mind this is the significant fix that should drive the next release, and anything else merged before then will also go along for the ride.

@tartley
Copy link
Owner

tartley commented Aug 25, 2022

@njsmith Invite in your inbox

@segevfiner
Copy link
Contributor Author

Edits by maintainers is enabled for this PR in case you want to change something by yourself to drive this to be merged, rather than asking me to do so.

I think PowerShell also enables VT handling, but it takes care to disable it before launching subprocesses so that they have to enable it by themselves. It might be a good idea to see how other libraries/tools that make use of this handle this correctly, as in, should we bother resetting it.

@njsmith
Copy link
Collaborator

njsmith commented Aug 25, 2022

I think PowerShell also enables VT handling, but it takes care to disable it before launching subprocesses so that they have to enable it by themselves.

Ah-ha, you're right. I wrote a little test script, that checks whether ANSI is enabled, and then unconditionally enables it: https://gist.github.com/njsmith/9cf1e9929947c6b7bb0ab330323beef4

When run from Powershell, it reports that ANSI is initially not enabled... but if I run it as a background job, so it and Powershell are running at the same time, then it reports that ANSI is initially enabled:

image

So that confirms that Powershell enables ANSI while it's running, but disables it before running a child process. Furthermore, it always resets the state after a child process exits, no matter what state the child process left things in. You can see here that if we run the program multiple times, it always starts with a "clean slate", even though the previous invocation left the console in ANSI mode:

image

I also tested under regular cmd.exe, and it turns out I misunderstood: I actually get the same results as with powershell.

The reason I thought cmd.exe set it unconditionally is that if you do os.system(""), then that does enable ANSI processing, for some mysterious reason. I thought it was because this effectively executes a do-nothing batch script through cmd.exe, but apparently not! It's mysterious. Not super important though.

Finally, all the above tests were with the standard built-in console. I also installed Microsoft's newer/fancier terminal app through the Windows store. When using this terminal, it looks like ANSI escapes are unconditionally enabled by default:

image

So, to sum up:

  • Powershell/cmd.exe with the default terminal: ANSI state is always reset by the shell, so it doesn't matter whether the program itself resets it on exit.
  • Powershell/cmd.exe with MS's newer terminal: ANSI is always enabled by default, so it doesn't matter whether we reset it on exit.
  • Running a python program directly (e.g. by double-clicking on it): this allocates a new console just for our program, and the console goes away when the program exits. So it doesn't matter whether we reset it on exit.

There are more complicated cases too, like if you have multiple programs running simultaneously in the same console. But in that case trying to reset the state is actually dangerous -- we might pull the rug out from underneath some other program, by disabling ANSI while the other program is still using it.

Also, we can't 100% reliably reset the state anyway (there's no guarantee that atexit handlers run, e.g. if python crashes, is killed, or the program calls os._exit()). A supervisor program like powershell/cmd.exe is really the only reliable place to handle this.

And finally, the worst case outcome here is... some program accidentally gets ANSI support where it wasn't expecting it. But who cares – are there really programs out there that are intentionally emitting ANSI escapes at the console and relying on them not being interpreted and rendering weird control character garbage instead? Seems very unlikely. And they're already broken by MS's own Terminal app.

So given all that, I'm pretty comfortable with simplifying this PR by dropping the atexit stuff.

@njsmith
Copy link
Collaborator

njsmith commented Oct 14, 2022

Finally looping back around to this...

I just pushed some fixes/cleanups to this PR branch. Changes:

  • Removed atexit machinery, as discussed above
  • Added a mock-based test (not very good, unfortunately, but it's hard to test this stuff without a real console)
  • Switched the mode switching code to working with the actual file handle passed to AnsiToWin32.__init__, instead of always working with STD_OUTPUT_HANDLE. In particular, this fixes the case where stderr is a console, but stdout is redirect to a file.

I also tested this manually on Windows 10 (native ansi console), Windows 10 (jupyter notebook, so no console), Windows 7 (classic console, w/o native ansi), and Linux, and verified that it was taking the code paths I expected in each case. So hopefully that should have caught any egregious errors when touching real devices, even if we don't have automated tests for them...

@njsmith
Copy link
Collaborator

njsmith commented Oct 14, 2022

Oh shoot, test failures on py2 because of missing contextlib.ExitStack. Please stand by...

@njsmith
Copy link
Collaborator

njsmith commented Oct 14, 2022

Ok, CI green! @segevfiner, @tartley any comments?

In my mind this is the significant fix that should drive the next release, and anything else merged before then will also go along for the ride.

I would like to sneak one more feature in if I can: a more stripped-down version of init, like colorama.just_fix_windows_console(). The motivation is, right now libraries like tqdm implicitly call colorama.init() at startup, but if you have multiple libraries doing this then you can end up with double-wrapping, and also end up with colorama's heuristics about when to strip ANSI from non-console output, which may not be what you want. So it'd be nice to have a minimalist version that does absolutely nothing except making sure that ANSI works as well as it can on Windows console.

@tartley
Copy link
Owner

tartley commented Oct 14, 2022

Acknowledged. Thank you. Sounds great!. I'll check it out in a few hours...

Copy link
Owner

@tartley tartley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great to me. Thanks to everyone who has contributed time, bug reports, knowledge and code along the way.

@tartley
Copy link
Owner

tartley commented Oct 16, 2022

@njsmith I'm happy to await your "stripped down init()" new feature. That sounds like a very sensible idea, and as you've seen, things don't move quickly here so there is no rush.

@tartley
Copy link
Owner

tartley commented Oct 16, 2022

@njsmith Having a quick look through existing issues and PRs, obviously there are some well-known problems with the existing init() (apologies! When I wrote this I thought it was a one-off script to make my tests display a nice green "OK" at the end) An alternative version of 'init' might be able to easily avoid repeating these mistakes:

Obviously I don't want to over-complicate your task. If these are trivially easy to accommodate, then great, but don't stress otherwise.

@segevfiner segevfiner deleted the win32-vt-processing branch October 16, 2022 22:25
@tartley
Copy link
Owner

tartley commented Oct 19, 2022

We have a release candidate on PyPI containing this PR. Thank you for your contributions! https://pypi.org/project/colorama/0.4.6rc1/

If nobody spots any problems with it, I'll push the final 0.4.6 tomorrow.

netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Nov 9, 2022
0.4.6 Current release
* tartley/colorama#139 Add alternative to 'init()',
  called 'just_fix_windows_console'. This fixes many longstanding problems
  with 'init', such as working incorrectly on modern Windows terminals, and
  wonkiness when init gets called multiple times. The intention is that it
  just makes all Windows terminals treat ANSI the same way as other terminals
  do. Many thanks the njsmith for fixing our messes.
* tartley/colorama#352 Support Windows 10's ANSI/VT
  console. This didn't exist when Colorama was created, and avoiding us
  causing havok there is long overdue. Thanks to segeviner for the initial
  approach, and to njsmith for getting it merged.
* tartley/colorama#338 Internal overhaul of package
  metadata declaration, which abolishes our use of the now heavily
  discouraged setuptools (and hence setup.py, setup.cfg and MANIFEST.in), in
  favor of hatchling (and hence pyproject.toml), generously contributed by
  ofek (author of hatchling). This includes dropping support Python3.5 and
  3.6, which are EOL, and were already dropped from setuptools, so this
  should not affect our users.
* tartley/colorama#353 Attention to detail award to
  LqdBcnAtWork for a spelling fix in demo06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0.4.6 Get merged for 0.4.6. release
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

9 participants