Man Page on Windows

[thecliguy]

I'm contemplating whether it would be possible make the man page available in Windows.

This is not a complete solution, these are just my initial thoughts exploring what would be required and how we might go about doing it... Any thoughts, feedback or suggestions would be welcome...

Converting the man page to a readable format for the Windows console

Since Windows doesn't have a manual reader, the man page would need to be converted to a format that can be rendered in the Windows console. This would have to be performed as part of the build process when there's a new release.

One option would be to simply convert it to plain text output. This conversion can be achieved as follows:

MANWIDTH=80 man ./ssh-audit.1 > ssh_audit_windows_man.txt

In Windows 10, the console is capable of interpreting ANSI escape sequences (also known as VT escape sequences). So another option would be to convert the man page to ANSI escape sequence formatted output, this would preserve any typographical emphasis that's present in the original man page, such as bold and underlined text. This conversion can be achieved as follows:

# * man outputs a backspace-overwrite sequence rather than an ANSI escape 
#   sequence.
# * 'MAN_KEEP_FORMATTING' preserves the backspace-overwrite sequence when 
#   redirected to a file or a pipe.
# * The 'ul' command converts the backspace-overwrite sequence to an ANSI escape 
#   sequence.

MANWIDTH=80 MAN_KEEP_FORMATTING=1 man ./ssh-audit.1 | ul > ssh_audit_windows_man.txt

Example of an ANSI escape sequence formatted man page on Windows 10

import os
os.system("color")

f = open('c:\\bitbucket\\ssh_audit_windows_man.txt', encoding="utf-8")
file_contents = f.read()
print (file_contents)
f.close()

Displaying the man page

Displaying the man page could perhaps be invoked using a command line parameter such as:

ssh-audit.exe --manual

Packaging the converted man page

Currently the Windows package is a standalone executable with no external dependencies. Ideally any solution that's adopted would preserve this.

Does anyone know of a way that the man page (in its converted format) could be embedded into the ssh-audit executable without having to ship an external text file?

[WSLUser]

Well ssh has a man page so I'd suggest looking there. Cygwin ssh would likely be your best bet.

[jtesta]

Does anyone know of a way that the man page (in its converted format) could be embedded into the ssh-audit executable without having to ship an external text file?

Not offhand. But PyInstaller's documentation should describe how to do this.

I'd be happy to this feature in, if you were interested in working on it.

[thecliguy]

@jtesta Let's say we do find a way to ship the man page (in its converted format) within the ssh-audit executable...

You'd need to add some steps to your Windows build process to produce the ANSI escape sequence formatted version of the man page (as described above). Are you OK with this?

I simply cannot find a way to easily render man pages on Windows in their native format. There's probably some library that exists for this purpose but I am opposed to adding any third party libraries into a project unless absolutely essential.

[jtesta]

You'd need to add some steps to your Windows build process to produce the ANSI escape sequence formatted version of the man page (as described above). Are you OK with this?

Yep. Every time the ssh-audit.1 file is updated, I could run a update_windows_man_page.sh script to create ssh-audit-windows-man-page.txt with the escape sequences preserved in it.

There's probably some library that exists for this purpose but I am opposed to adding any third party libraries into a project unless absolutely essential.

One of the original design goals of ssh-audit was to run with no external dependencies. The Windows build is a special case, however. Before PyInstaller is run, dependencies can be installed with pip. However, I'm pretty sure that wouldn't be needed. Colors can be enabled with the SetConsoleMode() Win32 API call (for an example, see https://github.com/rbsec/sslscan/blob/1530435c4eb7de0bc4eb47eda481497bf61b4d63/sslscan.c#L3654). After that, ANSI escape sequences work just fine (see https://github.com/rbsec/sslscan/blob/1530435c4eb7de0bc4eb47eda481497bf61b4d63/sslscan.h#L99).

I don't know off-hand how to call SetConsoleMode() from Python code, however...

[thecliguy]

What if instead of update_windows_man_page.sh producing a text file, it produced a python file, where it populates a string with the ANSI escape sequence formatted version of the man page?

WINDOWS_MAN_PAGE = '<ANSI Escape Sequence Formatted Value>'

Could ssh-audit then reference WINDOWS_MAN_PAGE as a variable so we don't have to package the man page as a text file?

[jtesta]

On Wed, 2021-01-20 at 14:58 -0800, thecliguy wrote: What if instead of update_windows_man_page.sh producing a text file, it produced a python file, where it populates a string with the ANSI escape sequence formatted version of the man page? WINDOWS_MAN_PAGE = '<ANSI Escape Sequence Formatted Value>' Could ssh-audit then reference WINDOWS_MAN_PAGE as a variable so we don't have to package the man page as a text file?

I was thinking that, actually. Then the code can do something like: ``` if sys.platform == "win32": import windows_man_page do_terminal_color_setup() # Somehow... print(WINDOWS_MAN_PAGE) ``` Two concerns about the above strategy: 1.) PyInstaller might not automatically package windows_man_page.py; it might need to be explicitly told (somehow), and 2.) pylint and/or flake8 might not like conditional imports.

…

-- Joseph S. Testa II Founder & Principal Security Consultant Positron Security

[thecliguy]

Here's another way we could approach it that avoids both of those potential problems...

Add WINDOWS_MAN_PAGE = '' to globals.py.

When performing a windows build, run update_windows_man_page.sh, it will perform a find and replace against globals.py, populating WINDOWS_MAN_PAGE with the ANSI escape sequence formatted version of the man page.

When a user invokes ssh-audit --manual, if WINDOWS_MAN_PAGE is not an empty string then print it to the console.

What do you think?

[jtesta]

What do you think?

Perfect!

[thecliguy]

I am making an assumption that PyInstaller automatically packages globals.py. Is that correct?

[jtesta]

I am making an assumption that PyInstaller automatically packages globals.py. Is that correct?

Yep.

[thecliguy]

@jtesta

I've written a function to:

• Determine if Windows supports ANSI escape sequences (support was first added in Windows 10 version 1511).
• If ANSI is supported then print the manual text, else strip the ANSI escape sequences and print a plain text version of the manual.

The function is invoked using the --manual or -m parameters.

I've a couple of implementation related questions for you...

1. Since this function is intended for use on Windows only, I think it makes sense to suppress its usage on other platforms. Are you happy doing this as follows in process_commandline?

        sopts = 'h1246M:p:P:jbcnvl:t:T:L'
        lopts = ['help', 'ssh1', 'ssh2', 'ipv4', 'ipv6', 'make-policy=', 'port=', 'policy=', 'json', 'batch', 'client-audit', 'no-colors', 'verbose', 'level=', 'timeout=', 'targets=', 'list-policies', 'lookup=']

        # The manual options are intended for use on Windows only.
        # Users of other operating systems should read the man page.
        if sys.platform == 'win32':
            sopts += 'm'
            lopts.append('manual')

2. Do you want to hide -m and --manual in usage on non-Windows platforms or just make its purpose clear by including Windows only in the explanation text, EG: -m, --manual print the man page (Windows only)?

[jtesta]

Do you want to hide -m and --manual in usage on non-Windows platforms or just make its purpose clear by including Windows only in the explanation text, EG: -m, --manual print the man page (Windows only)?

Perhaps `usage()` should explicitly tell the user that -m/--manual is for Windows only. And if it's used on a non-Windows OS, we print an error that it's not supported on this platform (I think this should answer your first question too).

[thecliguy]

@jtesta Thank you, that makes sense. I'll proceed on that basis.

In PACKAGING it says that the Windows package is built against Python 3.7.x. I see that PyInstaller (4.2 stable) now supports up to Python 3.9, see downloads. Do you want to continue building against Python 3.7.x or move to 3.9.x? Should I open a new issue for this?

[jtesta]

Do you want to continue building against Python 3.7.x or move to 3.9.x?

I'll switch to Python 3.9 for Windows builds, then.

Should I open a new PR for this?

To update PACKAGING? That wouldn't be necessary. I'll update it myself with tweaks after each time I package a new release.

[thecliguy]

@jtesta PR submitted, see #93.

[jtesta]

Thanks! I approved it, and changed a few things in 11e2e77. Namely, update_windows_man_page.sh now supports Cygwin only. This is because pyinstaller must be run from Windows, and we need the man program to filter the man-page into ANSI sequences.

Secondly, since colorama.init() does all the terminal color setup for us already, I removed all the direct Win32 API calls you included (I appreciate the work you put into that, though!). Unfortunately, the current colorama version ignores the underlining sequences, so we lose that in the translation (perhaps we can open an issue with them... update: there's already an open PR to support underlining: tartley/colorama#267).

And strangely, when I make a test build with pyinstaller 4.2 and Python 3.9.1 (the latest versions of both) and run with -m, Microsoft Defender improperly detects it as malware (!), specifically as "Trojan:Win32/Fuerboos.D!cl". I submitted this as a false positive to Microsoft, and will track this in a separate issue.

Other than that, if you're able to give it a test with my changes included, please let me know if you notice anything wrong! Thanks again!

[thecliguy]

Hi Joe, the code I had written was more verbose but served some specific purposes...

I was just looking at 11e2e77 and it does introduce a few problems...

• ANSI support was first introduced in Windows 10 version 1511. Therefore we can't just assume that ANSI support is available, we need to check if it's supported (using the Win32 API) and fall back to plain text in order to support older version of Windows that are still within active support. EG, this is how the output now looks on Windows Server 2012 R2, the escape sequence characters are being displayed as literal output:
  

• Another reason it's necesarry to have a plain text fallback is because if you redirect the output to a file or pipe it to another program (eg more), Windows suppresses ANSI support by design. Furthermore, if the output contains any unicode characters, the console's codepage must be capable of handling it. That's why I set the stdout encoding to utf8. This is what happens now if you try to redirect to more.
  (NB: I don't know why a non-unicode compatible code page only causes this issue when the output is not emitted directly to the console but that's the way it is)
  

Regarding update_windows_man_page.sh now only supporting Cygwin... Could we change this so it detects whether the platform is Cygwin or Linux? I'm not sure if that is possible, I haven't done any research to see if there's a way to identify if you're running Cygwin. I don't have Cygwin installed on my Windows machine at the moment, I haven't used it in years because I found the Windows Subsystem for Linux to be a lot better.

[thecliguy]

I see that update_windows_man_page.sh uses uname to identify Cygwin. So it should be quite easy to support both Cygwin and Linux. I can work on this if you're happy for me to do so?

[thecliguy]

A comment about underlined text...

colorama isn't needed for Windows to interpret and render underlined text when expressed as ANSI. In fact, colorama actually seems to be causing the problem...

To demonstrate, if you remove this:

ssh-audit/src/ssh_audit/ssh_audit.py

Lines 62 to 66 in 49cf91a

	try: # pragma: nocover
	from colorama import init as colorama_init
	colorama_init(strip=False) # pragma: nocover
	except ImportError: # pragma: nocover
	pass

And add os.system("") to windows_manual, then underlined text works fine:

[jtesta]

ANSI support was first introduced in Windows 10 version 1511. Therefore we can't just assume that ANSI support is available, we need to check if it's supported

According to this and this, version 1511 reached its end-of-life in October 2017. So we don't need to worry about Windows 10, it seems.

this is how the output now looks on Windows Server 2012 R2

I see that Server 2012 R2 is supported until late 2023. In that case, users on that platform can run with -n/--no-colors. I can add a regex to strip out the color codes in windows_manual() easily enough. Perhaps we can even do this automatically if we can detect Server 2012 at run-time...

Another reason it's necesarry to have a plain text fallback is because if you redirect the output to a file or pipe it to another program (eg more), Windows suppresses ANSI support by design.

3609461 fixes this. You can now run ssh-audit.py target | more and ssh-audit.py -m | more in both cmd.exe and Powershell with no problems.

Furthermore, if the output contains any unicode characters, the console's codepage must be capable of handling it.

This is also fixed in 3609461. Cygwin's man program was inserting unicode hyphens to break words. I modified update_windows_man_page.sh to translate the unicode hyphen to ascii hyphen. Try resetting your globals.py file and running the script again.

I see that update_windows_man_page.sh uses uname to identify Cygwin. So it should be quite easy to support both Cygwin and Linux. I can work on this if you're happy for me to do so?

Sure. I'm probably going to continue to use Cygwin when making official builds, but if you want to add support for another workflow for yourself, feel free!

To demonstrate, if you remove this: [...] And add os.system("") to windows_manual, then underlined text works fine:

Well that's pretty unexpected! I wonder why this os.system("") works...

I experimented with removing colorama for this new trick. We do indeed gain underlining functionality, but lose the ability to detect output re-direction & escape code filtering; escape codes end up being printed when piping output to more.

So the choice comes down to:

1. Remove colorama and add in lots of Win32 API calls, which gives us underlining functionality, or
2. Leave code as-is (which fixes all issues described above), and wait patiently for colorama to merge the PR which adds underlining.

I'd prefer waiting patiently for colorama so we have less code to maintain.

[jtesta]

@thecliguy Do you have access to a Server 2012 R2 machine? If so, can you show the output of platform.platform()? That seems to return detailed Windows version information in it. I can use that to automatically turn off colors.

[thecliguy]

@jtesta

Great, so we don't need to worry about unicode characters in the man page because you've stripped out unicode hyphens prior to export in update_windows_man_page.sh.

On Windows 10, now that you've changed colorama_init(strip=False) to colorama_init() ANSI escape characters are automatically removed when piping to another program (eg more) or redirecting to a file.

I agree about underlining, let's just wait for colorama to merge that PR.

So the last thing to do is engineer a way to automatically turn off colours in older version of Windows that lack ANSI support...

I believe that these are the versions of Windows released prior to Windows 10 that are still in support:

• Windows 8.1 - End of Extended Support: January 10, 2023
• Windows Server 2012 R2 - End of Extended Support: October 10, 2023

I feel confident in saying that Microsoft will not to backport ANSI support to either of the above, they are far too old in their respective product lifecycles to have any new features added. So we should automatically turn off colour on both of these Windows versions.

platform.platform() outputs the following on Windows Server 2012 R2:

Windows-2012ServerR2-6.3.9600-SP0

Perhaps it would be easier to use sys.getwindowsversion, EG:

if sys.getwindowsversion().major < 10:

---

Windows Server 2016

Windows Server 2016 was developed concurrently with Windows 10 (they share a
common code base) but it was released about a year later. Windows Server 2016 was
released to manufacturing bearing the version number of 10.0.14393 (same as Windows 10 Anniversary Update),
which is also known as version 1607.

We know that ANSI support was first added in Windows 10 version 1511, so in
theory if my detective work is correct this means that Windows Server 2016 has
always supported ANSI since its first proper release.

---

[jtesta]

Windows 8.1 - End of Extended Support: January 10, 2023

Ok, I'll look into how to identify Windows 8.1.

platform.platform() outputs the following on Windows Server 2012 R2: Windows-2012ServerR2-6.3.9600-SP0

Thanks!

Perhaps it would be easier to use sys.getwindowsversion, EG: if sys.getwindowsversion().major < 10:

What does this do on Windows Server 2016, though? I'd be wary of using this without testing first. Since I don't have Server 2016 handy, I'll probably go with platform.platform() to be safe. Thanks for the help!

[thecliguy]

For Windows 10, Windows Server 2016 and Windows Server 2019 the major part of the version number is 10. All previous versions of Windows use a major value below 10.

See Windows System Information: Operating System Version and Comparison of Microsoft Windows versions: Windows NT.

I currently only have access to Windows 10 and Windows Server 2012 R2 machines, here's what sys.getwindowsversion returns:

# Windows Server 2012 R2
sys.getwindowsversion(major=6, minor=3, build=9600, platform=2, service_pack='')

# Windows 10
sys.getwindowsversion(major=10, minor=0, build=18363, platform=2, service_pack='')

[jtesta]

Ok, I updated it to use sys.getwindowsversion(), since that doesn't require an additional import. Thanks for doing research on this!

[thecliguy]

I noticed that you placed the code for disabling colour within main:

ssh-audit/src/ssh_audit/ssh_audit.py

Lines 1029 to 1033 in 1b7cfbe

	# Disable color output on Windiows 8 and Windows Server 2012, as they are still supported by Microsoft (until Jan. 2023 and Oct. 2023, respectively); they do not support ANSI color codes. According to https://docs.microsoft.com/en-us/windows/win32/sysinfo/operating-system-version, the major versions of Server 2016, Server 2019, and Windows 10 are all 10.
	if (sys.platform == 'win32') and (sys.getwindowsversion().major < 10): # pylint: disable=no-member
	aconf.colors = False
	out.use_colors = False
	out.v("Disabling color output on this platform since it is not supported (Windows major version: %d)." % sys.getwindowsversion().major) # pylint: disable=no-member

This means that there is no colour output at all now on older versions of Windows, such as when performing a scan.

Older versions of Windows can handle colour output, they just don't understand ANSI. So you only need to suppress colour output within windows_manual.

[jtesta]

This means that there is no colour output at all now on older versions of Windows, such as when performing a scan. Older versions of Windows can handle colour output, they just don't understand ANSI. So you only need to suppress colour output within windows_manual.

They can do color in other ways? How? I thought older versions of Windows had no color support at all. The scan output uses ANSI color codes as well (see https://github.com/jtesta/ssh-audit/blob/1b7cfbec713a2d0cbc6ded2df8b768a4f745d9eb/src/ssh_audit/outputbuffer.py#L64 ). So we're already in an all-or-none situation. If there's another way to do color on Windows 8.1/Windows Server 2012 R2, I'm all ears!

[thecliguy]

They can do color in other ways? How? I thought older versions of
Windows had no color support at all.

Although Windows 10 version 1511 was the first version of Windows to support ANSI escape sequences, colour was previously supported by calling the Windows Console API.

Sorry about this... I just discovered something that I should have realised earlier... It appears that colorama must convert ANSI escape sequences into Windows Console API calls because when ssh-audit is packaged as an EXE or you run ssh-audit.py with colorama installed on Windows Server 2012 R2, it results in ANSI escape sequences to be interpreted correctly:

So actually, we don't need this:

ssh-audit/src/ssh_audit/ssh_audit.py

Lines 1029 to 1033 in 1b7cfbe

	# Disable color output on Windiows 8 and Windows Server 2012, as they are still supported by Microsoft (until Jan. 2023 and Oct. 2023, respectively); they do not support ANSI color codes. According to https://docs.microsoft.com/en-us/windows/win32/sysinfo/operating-system-version, the major versions of Server 2016, Server 2019, and Windows 10 are all 10.
	if (sys.platform == 'win32') and (sys.getwindowsversion().major < 10): # pylint: disable=no-member
	aconf.colors = False
	out.use_colors = False
	out.v("Disabling color output on this platform since it is not supported (Windows major version: %d)." % sys.getwindowsversion().major) # pylint: disable=no-member

In case colorama is not available then we should:

• Set a flag if the colorama import fails.
• If the flag is set then invoke this regular expression to fall back to plain text:
  

  ssh-audit/src/ssh_audit/ssh_audit.py

  Line 1015 in 1b7cfbe

  windows_man_page = re.sub(r'\x1b\[\d+?m', '', windows_man_page)

Apologies, this has been quite a journey of discovery... But I think this is the final revision.

Does this sound OK to you?

[thecliguy]

In case colorama is not available then we should:
* Set a flag if the colorama import fails.
* If the flag is set then invoke this regular expression to fall back to plain text:

How about this?

if not self.use_colors or not self.colors_supported:
  windows_man_page = re.sub(r'\x1b\[\d+?m', '', windows_man_page) 

[thecliguy]

Sorry, that example above is wrong - I need to take a break from this... 😩

Hopefully you get my point though.

[jtesta]

Sorry, that example above is wrong - I need to take a break from this... 😩

No problem. When you're ready to continue, I think the easiest path would be for you to test a patch against your Server 2012 machine, then submit a PR. Otherwise, I'm pretty much just guessing with changes on my end. No rush, though.

[thecliguy]

Thank you, there is just a small amount of work left to finish this.

I can visualise exactly what needs to be done, I will submit a PR tomorrow.

[thecliguy]

@jtesta PR submitted, see #95.

When I first wrote windows_manual my intention was to use the Win32 API to determine whether Windows supported ANSI or not. If ANSI was not supported, then I fell back to a plain text version of the man page. The function has since evolved and been simplified so it depends on the colorma library.

It isn't necessary to disable colour on older versions of Windows without native ANSI support because the colorama library converts ANSI into the appropriate win32 calls. Therefore I have made the following changes:

• Removed the code that disables colour for older versions of Windows.
• If the ssh-audit is invoked with a manual parameter (--manual / -m) and the colorama library was not imported then colour output is disabled, this results in falling back to a plain text version on the man page.

I've also:

• Updated update_windows_man_page.sh to support both Cygwin and Linux.
• Update the date to February 7, 2021 in ssh-audit.1. I forgot to do this in the original PR when I added -m and --manual to the man page.

---

Once Windows Server 2012 R2 and Windows 8.1 are out of support, all in-support versions of Windows should be capable of handling ANSI natively. When that time comes it would be worth thinking about whether the colorama library should be
dropped in favour of Windows native ANSI support. Something to think about in 2023... 🙂

---

[thecliguy]

@jtesta Sorry to bug you, just wondered if the PR and the explanation above all looks OK to you?

If you've got any questions please ask me anything...

[jtesta]

Merged. Thanks for helping me with this!

I'd like to get a release out the door soon. I'm just waiting on a block of hours to free up so I can focus on packaging everything up correctly. Hopefully that will be within the next few days!

Thanks again!