Add testdata for gherkin language

lexers/testdata/gherkin.actual

@@ -0,0 +1,88 @@
+@foo34 @bar # some comment
+Feature: proof of concept
+
+  In order to read Cucumber features faster and eaiser on Github 
+  As a Cucumber user
+  I want to have syntax highlighting for Gherkin
+  This is part of the narrative, even if I have the # symbol, it is still the narrative- not a comment
+
+  Background: this is multi-line description
+    do you like it?
+    Given cheese is good
+
+  Scenario Outline: this is a test 
+                    this the second line of the desctiption
+    Given I have a <var1> and some "string"
+    And the following table and some 'string'
+      | header 1  | header 2  | # here is a comment after a step table header
+      | cell 1-1  | cell 1-2  | # here is a comment after a step table row
+      | cell 2-1  | "cell 2-2" |
+
+    When I do <var2>
+    And use apostrophes then the feature's syntax still looks good
+    And here is a string with single and double quotes- "i'll be back" 
+    And here is a string with the opposite 'the quote is "Foo"'
+    And what about a var in a quote like so: "<var2>"
+    And what about a multiline string with a var in it like so:
+    """
+    Hello!
+    I am a multiple line string, often called a pystring.
+    I can have regular "double quotes" without a problem...
+    I can also sub in vars as well like: <var1> and <var2>
+    """
+
+    Then I should see something...
+    But not something else...
+
+  Examples:
+    | var1  |  var2  | # i am a comment
+    | foo   |  bar   | # so am i
+    | dog   |  food  |
+
+  Scenarios: some other examples with a description
+             and guess waht?!? I can have multilines as well!  # look at me, I'm a comment
+             Who would have thunk?
+    | var1  |  var2  |
+    | foo   |  bar   |
+    | dog   |  food  |
+
+  @some_tag @another_tag
+  Scenario: more examples
+            will follow after this multi-line description
+    Given some context# this is an inline comment
+    # This is a comment
+# So is this with no space at front...
+
+Funktionalität: Addition
+  Um dumme Fehler zu vermeiden
+  möchte ich als Matheidiot
+  die Summe zweier Zahlen gesagt bekommen
+
+  Szenariogrundriss: Zwei Zahlen hinzufügen
+    Gegeben sei ich habe <Eingabe_1> in den Taschenrechner eingegeben
+    Und ich habe <Eingabe_2> in den Taschenrechner eingegeben
+    Wenn ich <Knopf> drücke
+    Dann sollte das Ergebniss auf dem Bildschirm <Ausgabe> sein
+
+  Beispiele:
+    | Eingabe_1 | Eingabe_2 | Knopf | Ausgabe |
+    | 20        | 30        | add   | 50      |
+    | 2         | 5         | add   | 7       |
+    | 0         | 40        | add   | 40      |
+
+Feature: Addition
+  In order to avoid silly mistakes
+  As a math idiot 
+  I want to be told the sum of two numbers
+
+  Scenario Outline: Add two numbers
+    Given I have entered <input_1> into the calculator
+    * I have entered <input_2> into the calculator
+    When I press <button>
+    Then the result should be <output> on the screen
+
+  Examples:
+    | input_1 | input_2 | button | output |
+    | 20      | 30      | add    | 50     |
+    | 2       | 5       | add    | 7      |
+    | 0       | 40      | add    | 40     |

lexers/testdata/gherkin.expected

@@ -0,0 +1,246 @@
+[
+  {"type":"NameTag","value":"@foo34"},
+  {"type":"NameFunction","value":" "},
+  {"type":"NameTag","value":"@bar"},
+  {"type":"Comment","value":" # some comment"},
+  {"type":"NameFunction","value":"\n"},
+  {"type":"Keyword","value":"Feature:"},
+  {"type":"NameFunction","value":" proof of concept\n\n  In order to read Cucumber features faster and eaiser on Github \n  As a Cucumber user\n  I want to have syntax highlighting for Gherkin\n  This is part of the narrative, even if I have the # symbol, it is still the narrative- not a comment\n\n  "},
+  {"type":"Keyword","value":"Background:"},
+  {"type":"NameFunction","value":" this is multi-line description\n    do you like it?\n"},
+  {"type":"Keyword","value":"    Given "},
+  {"type":"NameFunction","value":"cheese is good\n\n  "},
+  {"type":"Keyword","value":"Scenario Outline:"},
+  {"type":"NameFunction","value":" this is a test \n                    this the second line of the desctiption\n"},
+  {"type":"Keyword","value":"    Given "},
+  {"type":"NameFunction","value":"I have a "},
+  {"type":"NameVariable","value":"\u003cvar1\u003e"},
+  {"type":"NameFunction","value":" and some \""},
+  {"type":"LiteralString","value":"string"},
+  {"type":"NameFunction","value":"\"\n    "},
+  {"type":"Keyword","value":"And "},
+  {"type":"NameFunction","value":"the following table and some 'string'\n"},
+  {"type":"Keyword","value":"      |"},
+  {"type":"LiteralString","value":" header 1"},
+  {"type":"Keyword","value":"  |"},
+  {"type":"LiteralString","value":" header 2"},
+  {"type":"Keyword","value":"  |"},
+  {"type":"Comment","value":" # here is a comment after a step table header"},
+  {"type":"Keyword","value":"\n      |"},
+  {"type":"LiteralString","value":" cell 1-1"},
+  {"type":"Keyword","value":"  |"},
+  {"type":"LiteralString","value":" cell 1-2"},
+  {"type":"Keyword","value":"  |"},
+  {"type":"Comment","value":" # here is a comment after a step table row"},
+  {"type":"Keyword","value":"\n      |"},
+  {"type":"LiteralString","value":" cell 2-1"},
+  {"type":"Keyword","value":"  |"},
+  {"type":"LiteralString","value":" \"cell 2-2\""},
+  {"type":"Keyword","value":" |\n"},
+  {"type":"NameFunction","value":"\n    "},
+  {"type":"Keyword","value":"When "},
+  {"type":"NameFunction","value":"I do "},
+  {"type":"NameVariable","value":"\u003cvar2\u003e"},
+  {"type":"NameFunction","value":"\n    "},
+  {"type":"Keyword","value":"And "},
+  {"type":"NameFunction","value":"use apostrophes then the feature's syntax still looks good\n    "},
+  {"type":"Keyword","value":"And "},
+  {"type":"NameFunction","value":"here is a string with single and double quotes- \""},
+  {"type":"LiteralString","value":"i'll be back"},
+  {"type":"NameFunction","value":"\" \n    "},
+  {"type":"Keyword","value":"And "},
+  {"type":"NameFunction","value":"here is a string with the opposite 'the quote is \""},
+  {"type":"LiteralString","value":"Foo"},
+  {"type":"NameFunction","value":"\"'\n    "},
+  {"type":"Keyword","value":"And "},
+  {"type":"NameFunction","value":"what about a var in a quote like so: \""},
+  {"type":"NameVariable","value":"\u003cvar2\u003e"},
+  {"type":"NameFunction","value":"\"\n    "},
+  {"type":"Keyword","value":"And "},
+  {"type":"NameFunction","value":"what about a multiline string with a var in it like so:\n    "},
+  {"type":"Keyword","value":"\"\"\""},
+  {"type":"LiteralString","value":"\n    Hello!\n    I am a multiple line string, often called a pystring.\n    I can have regular \"double quotes\" without a problem...\n    I can also sub in vars as well like: "},
+  {"type":"NameVariable","value":"\u003cvar1\u003e"},
+  {"type":"LiteralString","value":" and "},
+  {"type":"NameVariable","value":"\u003cvar2\u003e"},
+  {"type":"LiteralString","value":"\n    "},
+  {"type":"Keyword","value":"\"\"\""},
+  {"type":"NameFunction","value":"\n\n    "},
+  {"type":"Keyword","value":"Then "},
+  {"type":"NameFunction","value":"I should see something...\n    "},
+  {"type":"Keyword","value":"But "},
+  {"type":"NameFunction","value":"not something else...\n\n  "},
+  {"type":"Keyword","value":"Examples:\n    |"},
+  {"type":"NameVariable","value":" var1"},
+  {"type":"Keyword","value":"  |"},
+  {"type":"NameVariable","value":"  var2"},
+  {"type":"Keyword","value":"  |"},
+  {"type":"Comment","value":" # i am a comment"},
+  {"type":"Keyword","value":"\n    |"},
+  {"type":"NameVariable","value":" foo"},
+  {"type":"Keyword","value":"   |"},
+  {"type":"NameVariable","value":"  bar"},
+  {"type":"Keyword","value":"   |"},
+  {"type":"Comment","value":" # so am i"},
+  {"type":"Keyword","value":"\n    |"},
+  {"type":"NameVariable","value":" dog"},
+  {"type":"Keyword","value":"   |"},
+  {"type":"NameVariable","value":"  food"},
+  {"type":"Keyword","value":"  |\n"},
+  {"type":"NameFunction","value":"\n  "},
+  {"type":"Keyword","value":"Scenarios:"},
+  {"type":"NameFunction","value":" some other examples with a description\n             and guess waht?!? I can have multilines as well!"},
+  {"type":"Comment","value":"  # look at me, I'm a comment"},
+  {"type":"NameFunction","value":"\n             Who would have thunk?"},
+  {"type":"Keyword","value":"\n    |"},
+  {"type":"NameVariable","value":" var1"},
+  {"type":"Keyword","value":"  |"},
+  {"type":"NameVariable","value":"  var2"},
+  {"type":"Keyword","value":"  |"},
+  {"type":"NameFunction","value":"\n"},
+  {"type":"Keyword","value":"    |"},
+  {"type":"LiteralString","value":" foo"},
+  {"type":"Keyword","value":"   |"},
+  {"type":"LiteralString","value":"  bar"},
+  {"type":"Keyword","value":"   |"},
+  {"type":"NameFunction","value":"\n"},
+  {"type":"Keyword","value":"    |"},
+  {"type":"LiteralString","value":" dog"},
+  {"type":"Keyword","value":"   |"},
+  {"type":"LiteralString","value":"  food"},
+  {"type":"Keyword","value":"  |\n"},
+  {"type":"NameFunction","value":"\n  "},
+  {"type":"NameTag","value":"@some_tag"},
+  {"type":"NameFunction","value":" "},
+  {"type":"NameTag","value":"@another_tag"},
+  {"type":"NameFunction","value":"\n  "},
+  {"type":"Keyword","value":"Scenario:"},
+  {"type":"NameFunction","value":" more examples\n            will follow after this multi-line description\n"},
+  {"type":"Keyword","value":"    Given "},
+  {"type":"NameFunction","value":"some context"},
+  {"type":"Comment","value":"# this is an inline comment"},
+  {"type":"NameFunction","value":"\n"},
+  {"type":"Comment","value":"    # This is a comment"},
+  {"type":"NameFunction","value":"\n"},
+  {"type":"Comment","value":"# So is this with no space at front..."},
+  {"type":"NameFunction","value":"\n\n"},
+  {"type":"Keyword","value":"Funktionalität:"},
+  {"type":"NameFunction","value":" Addition\n  Um dumme Fehler zu vermeiden\n  möchte ich als Matheidiot\n  die Summe zweier Zahlen gesagt bekommen\n\n  "},
+  {"type":"Keyword","value":"Szenariogrundriss:"},
+  {"type":"NameFunction","value":" Zwei Zahlen hinzufügen\n"},
+  {"type":"Keyword","value":"    Gegeben sei "},
+  {"type":"NameFunction","value":"ich habe "},
+  {"type":"NameVariable","value":"\u003cEingabe_1\u003e"},
+  {"type":"NameFunction","value":" in den Taschenrechner eingegeben\n    "},
+  {"type":"Keyword","value":"Und "},
+  {"type":"NameFunction","value":"ich habe "},
+  {"type":"NameVariable","value":"\u003cEingabe_2\u003e"},
+  {"type":"NameFunction","value":" in den Taschenrechner eingegeben\n    "},
+  {"type":"Keyword","value":"Wenn "},
+  {"type":"NameFunction","value":"ich "},
+  {"type":"NameVariable","value":"\u003cKnopf\u003e"},
+  {"type":"NameFunction","value":" drücke\n    "},
+  {"type":"Keyword","value":"Dann "},
+  {"type":"NameFunction","value":"sollte das Ergebniss auf dem Bildschirm "},
+  {"type":"NameVariable","value":"\u003cAusgabe\u003e"},
+  {"type":"NameFunction","value":" sein\n\n  "},
+  {"type":"Keyword","value":"Beispiele:\n    |"},
+  {"type":"NameVariable","value":" Eingabe_1"},
+  {"type":"Keyword","value":" |"},
+  {"type":"NameVariable","value":" Eingabe_2"},
+  {"type":"Keyword","value":" |"},
+  {"type":"NameVariable","value":" Knopf"},
+  {"type":"Keyword","value":" |"},
+  {"type":"NameVariable","value":" Ausgabe"},
+  {"type":"Keyword","value":" |"},
+  {"type":"NameFunction","value":"\n"},
+  {"type":"Keyword","value":"    |"},
+  {"type":"LiteralString","value":" 20"},
+  {"type":"Keyword","value":"        |"},
+  {"type":"LiteralString","value":" 30"},
+  {"type":"Keyword","value":"        |"},
+  {"type":"LiteralString","value":" add"},
+  {"type":"Keyword","value":"   |"},
+  {"type":"LiteralString","value":" 50"},
+  {"type":"Keyword","value":"      |"},
+  {"type":"NameFunction","value":"\n"},
+  {"type":"Keyword","value":"    |"},
+  {"type":"LiteralString","value":" 2"},
+  {"type":"Keyword","value":"         |"},
+  {"type":"LiteralString","value":" 5"},
+  {"type":"Keyword","value":"         |"},
+  {"type":"LiteralString","value":" add"},
+  {"type":"Keyword","value":"   |"},
+  {"type":"LiteralString","value":" 7"},
+  {"type":"Keyword","value":"       |"},
+  {"type":"NameFunction","value":"\n"},
+  {"type":"Keyword","value":"    |"},
+  {"type":"LiteralString","value":" 0"},
+  {"type":"Keyword","value":"         |"},
+  {"type":"LiteralString","value":" 40"},
+  {"type":"Keyword","value":"        |"},
+  {"type":"LiteralString","value":" add"},
+  {"type":"Keyword","value":"   |"},
+  {"type":"LiteralString","value":" 40"},
+  {"type":"Keyword","value":"      |\n"},
+  {"type":"NameFunction","value":"\n"},
+  {"type":"Keyword","value":"Feature:"},
+  {"type":"NameFunction","value":" Addition\n  In order to avoid silly mistakes\n  As a math idiot \n  I want to be told the sum of two numbers\n\n  "},
+  {"type":"Keyword","value":"Scenario Outline:"},
+  {"type":"NameFunction","value":" Add two numbers\n"},
+  {"type":"Keyword","value":"    Given "},
+  {"type":"NameFunction","value":"I have entered "},
+  {"type":"NameVariable","value":"\u003cinput_1\u003e"},
+  {"type":"NameFunction","value":" into the calculator\n    "},
+  {"type":"Keyword","value":"* "},
+  {"type":"NameFunction","value":"I have entered "},
+  {"type":"NameVariable","value":"\u003cinput_2\u003e"},
+  {"type":"NameFunction","value":" into the calculator\n    "},
+  {"type":"Keyword","value":"When "},
+  {"type":"NameFunction","value":"I press "},
+  {"type":"NameVariable","value":"\u003cbutton\u003e"},
+  {"type":"NameFunction","value":"\n    "},
+  {"type":"Keyword","value":"Then "},
+  {"type":"NameFunction","value":"the result should be "},
+  {"type":"NameVariable","value":"\u003coutput\u003e"},
+  {"type":"NameFunction","value":" on the screen\n\n  "},
+  {"type":"Keyword","value":"Examples:\n    |"},
+  {"type":"NameVariable","value":" input_1"},
+  {"type":"Keyword","value":" |"},
+  {"type":"NameVariable","value":" input_2"},
+  {"type":"Keyword","value":" |"},
+  {"type":"NameVariable","value":" button"},
+  {"type":"Keyword","value":" |"},
+  {"type":"NameVariable","value":" output"},
+  {"type":"Keyword","value":" |"},
+  {"type":"NameFunction","value":"\n"},
+  {"type":"Keyword","value":"    |"},
+  {"type":"LiteralString","value":" 20"},
+  {"type":"Keyword","value":"      |"},
+  {"type":"LiteralString","value":" 30"},
+  {"type":"Keyword","value":"      |"},
+  {"type":"LiteralString","value":" add"},
+  {"type":"Keyword","value":"    |"},
+  {"type":"LiteralString","value":" 50"},
+  {"type":"Keyword","value":"     |"},
+  {"type":"NameFunction","value":"\n"},
+  {"type":"Keyword","value":"    |"},
+  {"type":"LiteralString","value":" 2"},
+  {"type":"Keyword","value":"       |"},
+  {"type":"LiteralString","value":" 5"},
+  {"type":"Keyword","value":"       |"},
+  {"type":"LiteralString","value":" add"},
+  {"type":"Keyword","value":"    |"},
+  {"type":"LiteralString","value":" 7"},
+  {"type":"Keyword","value":"      |"},
+  {"type":"NameFunction","value":"\n"},
+  {"type":"Keyword","value":"    |"},
+  {"type":"LiteralString","value":" 0"},
+  {"type":"Keyword","value":"       |"},
+  {"type":"LiteralString","value":" 40"},
+  {"type":"Keyword","value":"      |"},
+  {"type":"LiteralString","value":" add"},
+  {"type":"Keyword","value":"    |"},
+  {"type":"LiteralString","value":" 40"},
+  {"type":"Keyword","value":"     |\n"}
+]