Expose a getter for model tokens

[jefflevinson]

Originally, I was retrieving the data from the editor using this:
editor.getView().linesContent.children[3].children
where node [3] = div.view-lines
which gives a nicely structured list of tokens and data. However, my content got complicated enough that the content scrolled off of the view which is when I discovered it really only does give you what's in the view pane which wasn't obvious before. However, now I can't figure out how to get the data in the structured format for the non-visible portion of the editor. I've tried getModel and getView but it only comes back as raw text in those objects.

Thoughts?

[alexdima]

There are "@internal" API methods that don't show up in the monaco.d.ts. e.g. https://github.com/Microsoft/vscode/blob/8d43ec51ac3123a379342fe1a4e9e63cefda57e6/src/vs/editor/common/editorCommon.ts#L1776

interface IModel { 
...
    /**
     * Tokenize if necessary and get the tokenization result for the line `lineNumber`, as returned by the language mode.
     * @internal
     */
    getLineContext(lineNumber:number): ILineContext;
...
}

/**
 * @internal
 */
export interface ILineContext {
    getLineContent(): string;

    modeTransitions: ModeTransition[];

    getTokenCount(): number;
    getTokenStartIndex(tokenIndex:number): number;
    getTokenType(tokenIndex:number): string;
    getTokenText(tokenIndex:number): string;
    getTokenEndIndex(tokenIndex:number): number;
    findIndexOfOffset(offset:number): number;
}

We are not ready to commit on the API shape in this area, that's why it is "hidden".