Token Objects
Token objects are Struct objects, with a few helper methods.
-
type: a symbol, specifies the category of the token, such as :anchor, :set, :meta.
-
token: a symbol, the specific token for the type, such as :eol, :range, :alternation.
-
text: a string, the text of token, such as '$', 'a-z', '|'.
-
ts: an integer, the start offset within the entire expression (measured in chars/graphemes).
-
te: an integer, the end offset within the entire expression (measured in chars/graphemes).
-
level: an integer, the group nesting level at which the token appears.
-
set_level: an integer, the character set nesting level at which the token appears.
-
conditional_level: an integer, the conditional expression nesting level at which the token appears.
-
next: return the next token in the list, or nil if it's the last one.
-
previous: returns the previous token in the list, or nil if it's the first one.
-
offset: returns an array with the start and end offsets of the token within the root expression.
-
length: returns the length of the text of the token.
-
to_h: returns the token's attributes as a Hash object.