Added an optional callback OnToken

[schaakverslaafd]

Types of Changes

Prerequisites

Please make sure you can check the following two boxes:

• I have read the CONTRIBUTING document
• My code follows the code style of this project

Contribution Type

What types of changes does your code introduce? Put an x in all the boxes that apply:

• Bug fix (non-breaking change which fixes an issue, please reference the issue id)
• New feature (non-breaking change which adds functionality, make sure to open an associated issue first)
• Breaking change (fix or feature that would cause existing functionality to change)
• My change requires a change to the documentation
• I have updated the documentation accordingly (NOT YET)
• I have added tests to cover my changes
• All new and existing tests passed

Description

Added an OnToken event to the htmlParser raised when a new html token is consumed. This event passes the html token and a range in the source text corresponding to the token.
I didn't update documentation (yet). Provided this pull request to discuss issue #754.

[FlorianRappl]

Alright we definitely need to see two benchmarks here (on a large HTML document):

1. Old code (no modification)
2. With OnToken (does not need to be provided; the performance should roughly remain the same for the case that no OnToken callback was provided)

Could this be done? Thanks for your efforts!

[schaakverslaafd]

Hi,

Sorry for a late reponse.

Two benchmarks from parsing page.html from the benchmark project.

Benchmark for old code

Benchmark with OnToken

using System.IO;
using AngleSharp.Html.Parser;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;

namespace AngleSharp.Benchmarks
{
    [MemoryDiagnoser, GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByParams), ShortRunJob]
    public class OnTokenBenchmark
    {
        private static readonly HtmlParser angleSharpParser = new HtmlParser();
        private string pageContent = "";

        [GlobalSetup]
        public void GlobalSetup()
        {
            pageContent = File.ReadAllText("page.html");
        }

        [Benchmark]
        public void AngleSharp()
        {
            angleSharpParser.ParseDocument(pageContent);
        }
    }
}

[FlorianRappl]

Great thanks for the efforts @schaakverslaafd - can you update your branch (sync with devel)? Then we are ready to merge!

[schaakverslaafd]

Great!
I merged the branches. My editor messed up the spacing a bit. Let me know if that's an issue.

(The original solution had to be changed a bit because StructHtmlToken is passed by ref. New benchmarks did not show a significant difference.)

[schaakverslaafd]

Alright! The space changes are removed.

[FlorianRappl]

LGTM - thanks for all your efforts 🚀 !