Convert weird chars into their proper version

[PrzemyslawKlys]

Summary of the new feature

I saw this reddit thread https://www.reddit.com/r/PowerShell/comments/9rlj00/why_does_visualstudio_code_keep_borking_my_scripts/ and it reminded me having those problems as well. Basically when you copy/paste code from different websites it looks good but chars like - or ' or " often have weird version of it and it looks like gibberish after you run it.

On website:

Set-ADUser $user –replace @{EmployeeType=”$newemployeetype”} -WhatIf

When pasted it looks almost the same...

But when run I often see complains and output displays this:

Set-ADUser $user –replace @{EmployeeType=�$newemployeetype�} -WhatIf

It's also visible when you do something in Word (I know I know), Outlook or other software like that.

In word:

After paste

After run:

This is related to how polish languages treats double quotes. But I am sure it's similar on a lot of websites, wrong templates, old outdated Microsoft resource websites?

Also right now VSCode doesn't regognize this as a problem until it's too late. Normally on errors it would show "red" file but here.. everything is great.

As opposed to:

Proposed technical implementation details (optional)

I had this conversation with @tylerl0706 on twitter: https://twitter.com/TylerLeonhardt/status/1055859999086235648 and one of proposed implementations was to utilize format on paste, format on save, format on... where it would magically replace those weird chars into their equivalent versions.

[bergmeister]

Thanks for raising it again and providing details but this is a duplicate of #981 in terms of detecting those characters and possibly converting them with a suggested correction. There is already a VS-Code extension here that highlights those characters. VS-Code specific actions such as format on paste should be raised and implemented in https://github.com/PowerShell/vscode-powershell and/or https://github.com/PowerShell/PowerShellEditorServices

[msftrncs]

Actually this is an area where EditorSyntax needed some enhancement. A PR is now waiting to be merged, and this this really isn't an issue, as PowerShell does accept those quotes.

PowerShell/EditorSyntax#141

PR: PowerShell/EditorSyntax#173

Some characters look like gibberish because of a encoding issue in the terminal, or a lack of those characters being available in the font. They have no affect on how PowerShell executes as long as the encoding is correct when PowerShell reads the script.