Refactor/Rewrite analyze_text

[Anteru]

analyze_text is a source of problems and has an impossible task to do -- it's supposed to locally decide how well a lexer matches a given piece of code, which is then used to make a global decision among all lexers. This is leading to tons of issues, and we have a large number out outstanding PRs related to it:

• fix ecl analyze text logic #1610
• Fix mysql analyze text logic #1593
• Fix CBM BASIC V2 analyze text logic #1607
• Jasmin min score for analyse text #1619
• MIME lexer text analyser error #1692

as well as issues like (those have been closed as they are all related to this issue):

• guess_lexer returns MIMELexer for Python code #1584
• Priority-based lexer selection in get_lexer_by_name et al. #1603
• Incorrect Java file lexer guess #1563
• guess_lexer returns pygments.lexers.XmlLexer instead of C# or Java parser #449
• guess_lexer returns pygments.lexers.MatlabLexer instead of C# parser #450
• guess_lexer returns pygments.lexers.MatlabLexer instead of C# parser #451
• Lexer for CMakeLists.txt is guessed incorrectly #523
• Speeding up get_*() and guess_*() #1147
• Inconsistently styled space after prompt #1083

There are two "right" solutions to this, and they can be implemented incrementally. The first one is to default analyze_text to only invoke lexers which support the given file extension. This will dramatically improve the accuracy, as there will be only 2-3 lexers that need to get coordinated instead of having any lexer participate (and some lexers do accept nearly everything.) This can be implemented right now, but it requires touching every single lexer and clean up the weights relative to other candidates.

The second step -- which is more involved -- is to turn to machine learning to train a model that can make global decisions by knowing all languages and being able to weight the probabilities globally. This would be the "endgame" solution for analyze_text, with the hope that we can have a reinforcement process in place to teach the model whenever an issue occurs. That's a much larger task though, similar in scope to https://github.com/github/linguist

[alanhamlett]

Re-using linguist's training would be ideal for the machine learning second solution.