(yaml) improve tag support; add verbatim tags #2487

pplantinga · 2020-04-09T21:21:55Z

Resolves #2486

src/languages/yaml.js

joshgoebel

Oh it's a secondary handle vs a named handle. I love YAML. 🙄

src/languages/yaml.js

joshgoebel · 2020-04-09T22:28:53Z

Can we add a markup file with a smattering of actual examples of tags in use and add the missing characters to the regex?

pplantinga · 2020-04-10T21:12:00Z

All characters have been added, as well as a few tests. Ready for review.

joshgoebel · 2020-04-10T23:17:51Z

See my pending review on the removal of !!.

pplantinga · 2020-04-12T12:34:09Z

I can't find the pending review. Where can I see it?

src/languages/yaml.js

joshgoebel · 2020-04-13T05:56:54Z

Do you see it now?

joshgoebel · 2020-04-14T07:38:22Z

src/languages/yaml.js

@@ -10,6 +10,10 @@ Category: common, config
 export default function(hljs) {
  var LITERALS = 'true false yes no null';

+  // YAML spec allows non-reserved characters in tags
+  var NON_EX_CHARS = '\\w#;/?:@&=+$,.~*\\\'()[\\]'


What is "EX"?

It was originally NON_EXCLAMATION_CHARS but that was too long. Maybe NON_PREFIX_CHARS is better?

It wasn't too long.

Wait, are they non prefix character or prefix characters?

You're right, prefix is confusing (here I meant '!' as prefix). Going back to exclamation.

joshgoebel · 2020-04-14T07:40:15Z

src/languages/yaml.js

@@ -72,11 +76,11 @@ export default function(hljs) {
      },
      { // local tags
        className: 'type',
-        begin: '!' + hljs.UNDERSCORE_IDENT_RE,
+        begin: '!' + YAML_TAG_RE,


Let's change it to YAML_TAG_PREFIX - unless you tell me I'm mistaken? And then I think this looks ok.

Tag prefixes are available in YAML but I'm not sure they work exactly the way you think they do. Basically, at the top of a YAML document you can define a tag prefix which is basically just a shorthand for a longer tag, e.g.:

%TAG !e! tag:example.com,2000:app/ --- !e!foo "bar"

Which converts !e!foo to !<tag:example.com,2000:app/foo> during parsing, according to the spec.

I'm open to other names for this variable, but I'm not sure YAML_TAG_PREFIX is appropriate, since we're matching the whole tag.

https://yaml.org/spec/1.2/spec.html#id2783273

I'm thinking "local tag prefix" which start with ! from my read but globals don't have ! evidentaly. Is your intention here that this is matching tags as well as tag prefixes, yes?

Ie, we're matching !tagname! with this also, yes?

After reading more carefully, I don't think it works the way I thought it did either. Most of what I've read so far merely relates to the %TAG directive, which allows three tag prefixes, !, !!, and ! \w+ !, then it allows a replacement that includes an URI characters.

However, this doesn't define what is allowed in full tags. I actually can't find what is allowed in full tags in this document, but given how PyYAML uses the tags, it seems they can include at least / and : and .

I propose allowing all URI characters in tags since this is how other syntax highlighters are doing it (e.g. vim, although I don't think vim allows square brackets).

However, this doesn't define what is allowed in full tags.

What are you calling a "full tag" here?

A tag is only a handle and a prefix unless I'm missing something.

Then there are primary handles !, secondary !! and named handles:

[92] | c-named-tag-handle | ::= | “!” ns-word-char+ “!”

Isn't that it? I feel like this rule (as currently written) is also going to capture named tag handles - and that might be ok... if not we should perhaps add matchers for the 3 tag handle variants.

If this extra complexity is to deal with tag handles vs prefixes would it be simpler to have a sep rule for the handles? entirely? !, !!, and ! \w+ ! as you already said. Would that simplify the prefix rule?

To clarify I by "full tag" I meant !yaml!str where the tag prefix (or tag handle) refers to !yaml!. What is allowed to come after the tag prefix (in this example: str) is not very clearly defined in the spec. The only thing I can find is in section 3.2.1.2 Tags and it goes like this:

local tags are restricted to the URI character set and use URI character escaping.

And it goes on to describe how different languages will use the tags differently (e.g. Java tags use .). So full URI character set is what I recommend for tags (the part after the prefix). If you want to do it entirely properly, probably what would be required is three rules for the three valid prefixes, plus excluding exclamation points from the character set.

joshgoebel · 2020-04-17T16:57:17Z

One of your tests is off.

pplantinga · 2020-04-17T17:01:28Z

Are the regexes applied in order? The failing test splits !foo!bar into two separate tags, which perhaps can only be fixed if the named tag regex is applied before the local tag regex. Otherwise I can just go back to including ! in tags which would capture named tags too

joshgoebel · 2020-04-17T17:22:35Z

Let me know when you're happy and I will pull this local for one last review and maybe a tweak or two and then try to get this merged.

joshgoebel · 2020-04-17T17:23:00Z

Are the regexes applied in order?

They are.

pplantinga · 2020-04-17T17:26:08Z

Okay, I reverted adding named tag handles, since I'm not quite familiar enough with how the regexes are applied. The fact that they are captured by the local tag regex should be enough in practice, its just not quite as explicit as it could be. I'm happy with this request, feel free to pull and test and make changes.

… tag handles

joshgoebel · 2020-04-21T23:15:00Z

All I did was add verbatim tags since I came across them while doing some final reading on this. Waiting for a newer changelog in another PR then I'll merge this. Right now its gonna wait until 10.1 anyways.

@objcMembers

* (docs) Add Chaos to supported languages (highlightjs#2510) * fix(parser) fixes sublanguage with no rule matches (highlightjs#2506) * fix(parser) fixes sublanguage with no rule matches Resolves highlightjs#2504. * (chore) Add ESLint config and clean up the major stuff (highlightjs#2503) * (chore) eslint:recommended * (chore): eslint_standard * relax eslint rules for language grammars (to discourage rewriting them in one fell swoop; I'd rather have the blame history intact) * remove extra escaping * clean up variables * more camelcase * (docs) Add Visual Basic for Applications (VBA) to supported languages (highlightjs#2512) * (yaml) improve tag support; add verbatim tags (highlightjs#2487) * YAML parse non-word characters as part of tags * adds support for verbatim tags Co-authored-by: Josh Goebel <me@joshgoebel.com> * fix(javascript/typescript): lambda with parens in parameters fails (highlightjs#2502) * fix(javascript/typescript): lambda with parens in parameters fails - Fixes both JavaScript and TypeScript grammars Fixes samples like: const bad = ((a, b) => [...a, b]); sides.every((length,width=(3+2+(4/5))) => length > 0 ); This is done by counting parens in the regex that finds arrows functions. Currently we can only handle 2 levels of nesting as shown in the second example above. * allow much richer highlighting inside params * improve highlighting inside arguments on typescript * enh(cpp): Improve highlighting of unterminated raw strings PR highlightjs#1897 switched C++ raw strings to use backreferences, however this breaks souce files where raw strings are truncated. Like comments, it would be preferable to highlight them. - Add `on:begin` and `on:end` to allow more granular matching when then end match is dynamic and based on a part of the begin match - This deprecates the `endSameAsBegin` attribute. That attribute was a very specific way to solve this problem, but now we have a much more general solution in these added callbacks. Also related: highlightjs#2259. Co-authored-by: Josh Goebel <me@joshgoebel.com> * (chore) C-like uses the new END_SAME_AS_BEGIN mode * (chore) Ruby uses END_SAME_AS_BEGIN mode/rule * (parser) make END_SAME_AS_BEGIN a function helper Adds a mode helper to replace the deprecated `endSameAsBegin` attribute. The first match group from the begin regex will be compared to the first match group from the end regex and the end regex will only match if both strings are identical. Note this is more advanced functionality than before since now you can match a larger selection of text yet only use a small portion of it for the actual "end must match begin" portion. * (pgsql) add test for $$ quoting existing behavior - even if that existing behavior is questionable - the ending span should really close before the $$, not after Fixing this would involve delving into the sublanguage behavior and I'm not sure we have time to tackle that right this moment. * (chore) pgsql uses END_SAME_AS_BEGIN mode/rule now also * (docs) rename to `mode_reference`; docs for callbacks - I can never find this file because it's name didn't fully match. - rename callbacks to `on:begin` and `on:end` * prevented setter keyword conflicting with setTimeout|setInterval and highlighted them (highlightjs#2514) (highlightjs#2515) * fix(javascript) prevent setter keyword 'set' conflicting with setTimeout|setInterval (highlightjs#2514) * enh(javascript) setTimeout|setInterval now highlighted (highlightjs#2514) * enh (javascript) clearInterval and clearTimeout now highlighted * add keywords to TypeScript also * (docs) add TLDR instructions for building and testing * (dev) improve developer tool UI * (parser) Build common EMCAscript foundation Builds a common keyword foundation for any grammar that is building on top of JavaScript: - LiveScript - CoffeeScript - TypeScript Also uses this common foundation for JS itself. * (parser) Adds SHEBANG mode * (yaml) Add support for inline sequences and mappings (highlightjs#2513) * Use containers to match inline sequences and mappings * Add string type for inside inline elements * Handle nested inline sequences and mappings * Disallow all braces brackets and commas from strings inside inline mappings or sequences * clean up implementation * feed the linter Co-authored-by: Josh Goebel <me@joshgoebel.com> * [enh] Add `OPTIMIZE:` and `HACK:` to comment doctags * (build) browser build is CommonJS and IIFE, no more AMD (highlightjs#2511) * (build) browser build is CommonJS and IIFE (global) now * (build) dropping support for AMD, which we never truly supported properly in the first place * (build) add test to make sure browser build works as commonJS module Resolves highlightjs#2505 * fix(parser) Fix freezing issue with illegal 0 width matches (highlightjs#2524) * fix[parser] add edge case handle for illegal 0 width matches * add last ditch catch all that tries to detect other uncaught freezes * (docs) added unicorn-rails-log as an 3rd-party language (highlightjs#2528) - (docs) Add syntax highlighting for Rails Unicorn logging to supported languages. * (docs) fix supported languages link: it moved again! (highlightjs#2533) * fix(ts/js) use identifier to match potential keywords (highlightjs#2519) - (parser) Adds `keywords.$pattern` key to grammar definitions - `lexemes` is now deprecated in favor of `keywords.$pattern` key - enh(typescript) use identifier to match potential keywords, preventing false positives - enh(javascript) use identifier to match potential keywords, preventing false positives * fix(javascript) fix regex inside parens after a non-regex (highlightjs#2531) * make the object attr container smarter * deal with multi-line comments also * comments in any order, spanning multiple lines Essentially makes the object attr container much more sensitive by allowing it to look-ahead thru comments to find object keys - and therefore prevent them from being incorrectly matched by the "value container" rule. * (parser) Add hljs.registerAlias() public API (highlightjs#2540) * Add hljs.registerAlias(alias, languageName) public API * Add .registerAlias() test * enh(cpp) add `pair`, `make_pair`, `priority_queue` as built-ins (highlightjs#2538) * (fix) `fixMarkup` would rarely destroy markup when `useBR` was enabled (highlightjs#2532) * enh(cpp) Recognize `priority_queue`, `pair` as containers (highlightjs#2541) * chore: rename `registerAlias` to `registerAliases` Plural form is clearly better as it's not surprising to the reader to see it being passed an array - where as the singular form might have been. Meanwhile it's also easy to assume that it also supports arrays of any size - including an array with a singular alias. The fact that it can magically accept a string as the first argument might not be obvious, but we document it and even if one didn't know this they could still use the array form of the API without any issue by passing a one item array. * (swift) @objcMembers not completely highlighted (highlightjs#2543) * Fixed @objcMembers in Swift Would match `@objc` first, and the `Members` part would be unhighlighted * Update CHANGES.md * Update swift.js * (docs) add OCL to list of supported languages (highlightjs#2547) * (docs) Add Svelte to list of supported languages (highlightjs#2549) * enh(dart) Add `late` and `required` keywords, and `Never` built-in type (Dart 2.9) (highlightjs#2551) * Add new Dart 2.9 keywords for Null Safety language feature * enh(erlang) add support for underscore separators in numeric literals (highlightjs#2554) * (erlang) add support for underscore separators in numeric literals * (erlang) add tests * (docs) add Jolie to Supported Languages (highlightjs#2556) * (parser/docs) Add jsdoc annotations and TypeScript type file (highlightjs#2517) Adds JSDoc annotations and a .tsconfig that allows TypeScript to be run in it's "allowJS" mode and apply type and sanity checking to JavaScript code also. See Type Checking JavaScript Files. I've been using TypeScript a lot lately and finding it very beneficial and wanted to get those same benefits here but without converting the whole project to TypeScript. It was rough at the beginning but now that this is finished I think it's about 80%-90% of the benefits without any of the TS compilation pipeline. The big difference in being JSDoc for adding typing information vs inline types with TypeScript. Should be super helpful for maintainers using an editor with tight TypeScript integration and the improved docs/comments should help everyone else. - Adds types/index.d.ts to NPM build (should be useful for TypeScript peeps) - Improves documentation of many functions - Adds JSDoc annotations to almost all functions - Adds JSDoc type annotations to variables that can't be inferred - Refactors a few smaller things to allow the TypeScript compiler to better infer what is happening (and usually also made the code clearer) * (parser) highlightBlock result key `re` => `relevance` (highlightjs#2553) * enh(handlebars) Support for sub-expressions, path-expressions, hashes, block-parameters and literals (highlightjs#2344) - `htmlbars` grammar is now deprecated. Use `handlebars` instead. A stub is included so that anyone literally referencing the old `htmlbars` file (say manually requiring it in Node.js, etc) is still covered, but everyone should transition to `handlebars` now. * fix(typescript) Add missing `readonly` keyword (highlightjs#2562) * (docs) Mention `c` is a possible class for C (highlightjs#2577) * fix(groovy) strings are not allowed inside ternary clauses (highlightjs#2565) * fix(groovy) strings are not allowed inside ternary clauses * whitespace can also include tabs * Update @typescript-eslint/parser to the latest version 🚀 (highlightjs#2575) * chore(package): update @typescript-eslint/parser to version 3.0.0 * chore(package): update lockfile package-lock.json Co-authored-by: greenkeeper[bot] <23040076+greenkeeper[bot]@users.noreply.github.com> Co-authored-by: Josh Goebel <me@joshgoebel.com> * Update @typescript-eslint/eslint-plugin to the latest version 🚀 (highlightjs#2576) * chore(package): update @typescript-eslint/eslint-plugin to version 3.0.0 * chore(package): update lockfile package-lock.json Co-authored-by: greenkeeper[bot] <23040076+greenkeeper[bot]@users.noreply.github.com> Co-authored-by: Josh Goebel <me@joshgoebel.com> * (parser) properly escape ' and " in HTML output (highlightjs#2564) * escape quotes also in final HTML output * [style] update test coding style * update markup tests with new escaping This shouldn't be a security issue -- we've always escaped double quotes inside of HTML attribute values (where they could be used to break out of context) - and we've always used double quotes for enclosing attribute values. This just goes all the way and now properly escapes quotes everywhere. Better safe than sorry. * (docs) add changelog entry for last PR * add nnfx theme (highlightjs#2571) * (themes) Add new lioshi theme (highlightjs#2581) * Added Cisco Command Line to SUPPORTED_LANGUAGES.md (highlightjs#2583) * (themes) add `nnfx-dark` theme (highlightjs#2584) * enh(protobuf) Support multiline comments (highlightjs#2597) * enh(java) added support for hexadecimal floating point literals (highlightjs#2509) - Added support for many additional types of floating point literals - Added related tests There still may be a few gaps, but this is a pretty large improvement. Co-authored-by: Josh Goebel <me@joshgoebel.com> * (chore) Update issue templates (highlightjs#2574) Co-authored-by: Vladimir Jimenez <allejo@me.com> * enh(toml)(ini) Improve parsing of complex keys (highlightjs#2595) Fixes: highlightjs#2594 * (chore) add `.js` extension to import statements (highlightjs#2601) Adds file extensions to all import specifiers in ./src/ files. This is useful to run the files straight from source with a web browser , Node.js ESM or Deno. - Also add eslint rules regarding extensions for imports * enh(dart) highlight built-in nullable types (highlightjs#2598) * Dart: allow built-in nullable types with trailing ? to be highlighted * enh(csharp) highlight generics in more cases (highlightjs#2599) * (chore) fix tiny style issues, add linting npm task - fixes tiny style issues - adds `npm run lint` for linting the main library source (not languages which are still much messier) * (chore) bump dev dependencies * (chore) upgrade some dev stuff to newer versions * bump v10.1.0 * (chore) bump copyright * (chore) more import below metadata comment Co-authored-by: M. Mert Yıldıran <mehmetmertyildiran@gmail.com> Co-authored-by: Josh Goebel <me@joshgoebel.com> Co-authored-by: Hugo Leblanc <dullin@hololink.org> Co-authored-by: Peter Massey-Plantinga <plantinga.peter@gmail.com> Co-authored-by: David Benjamin <davidben@google.com> Co-authored-by: Vania Kucher <dev.kucher@gmail.com> Co-authored-by: SweetPPro <sweetppro@users.noreply.github.com> Co-authored-by: Alexandre ZANNI <16578570+noraj@users.noreply.github.com> Co-authored-by: Taufik Nurrohman <t.nurrohman77@gmail.com> Co-authored-by: Lin <50829219+Linhk1606@users.noreply.github.com> Co-authored-by: nicked <nicked@gmail.com> Co-authored-by: Nicolas Homble <nhomble@terpmail.umd.edu> Co-authored-by: Ryandi Tjia <ryandi.tjia@me.com> Co-authored-by: Sam Rawlins <srawlins@google.com> Co-authored-by: Sergey Prokhorov <seriy.pr@gmail.com> Co-authored-by: Brian Alberg <brian@alberg.org> Co-authored-by: Nils Knappmeier <github@knappi.org> Co-authored-by: Martin <7252614+Lhoerion@users.noreply.github.com> Co-authored-by: Derek Lewis <DerekNonGeneric@inf.is> Co-authored-by: greenkeeper[bot] <23040076+greenkeeper[bot]@users.noreply.github.com> Co-authored-by: Jim Mason <jmason@ibinx.com> Co-authored-by: lioshi <lionel.fenneteau@gmail.com> Co-authored-by: BMatheas <65114274+BMatheas@users.noreply.github.com> Co-authored-by: Pavel Evstigneev <pavel.evst@gmail.com> Co-authored-by: Vladimir Jimenez <allejo@me.com> Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com> Co-authored-by: TupikovVladimir <vladimir.tupikov@devexpress.com>

joshgoebel reviewed Apr 9, 2020

View reviewed changes

src/languages/yaml.js Outdated Show resolved Hide resolved

joshgoebel reviewed Apr 9, 2020

View reviewed changes

src/languages/yaml.js Outdated Show resolved Hide resolved

joshgoebel reviewed Apr 13, 2020

View reviewed changes

src/languages/yaml.js Show resolved Hide resolved

joshgoebel reviewed Apr 14, 2020

View reviewed changes

joshgoebel added this to the 10.1 milestone Apr 21, 2020

pplantinga and others added 11 commits April 21, 2020 19:02

YAML parse non-word characters as part of tags highlightjs#2486

017f978

Add message for highlightjs#2486 to CHANGES.md

6a5668c

YAML add all characters for matching tags, plus tests

bfcbbd8

YAML tags double escape characters

53ce6e0

YAML add back data type tags and disallow "!" in first character of tags

1992e51

YAML clarify tag definition

a14893b

Add named tag handles and clarify definition

7e92cd0

YAML fix order of applying regexes for tags

3466812

Revert adding named tag handles, since they are captured by the local…

7939322

… tag handles

Add named tag again

6ca7314

fixup comments, add verbatim tags

05471b9

joshgoebel force-pushed the yaml_highlight_fix branch from 15a45b4 to 05471b9 Compare April 21, 2020 23:03

joshgoebel changed the title ~~YAML parse non-word characters as part of tags #2486~~ (yaml) improve tag support; add verbatim tags Apr 21, 2020

joshgoebel approved these changes Apr 21, 2020

View reviewed changes

joshgoebel added 2 commits April 27, 2020 01:15

Merge branch 'master' into yaml_highlight_fix

7e4d779

Update CHANGES.md

677bf3a

joshgoebel approved these changes Apr 27, 2020

View reviewed changes

joshgoebel merged commit 7502e42 into highlightjs:master Apr 27, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(yaml) improve tag support; add verbatim tags #2487

(yaml) improve tag support; add verbatim tags #2487

pplantinga commented Apr 9, 2020 •

edited by joshgoebel

joshgoebel left a comment

joshgoebel commented Apr 9, 2020

pplantinga commented Apr 10, 2020

joshgoebel commented Apr 10, 2020

pplantinga commented Apr 12, 2020

joshgoebel commented Apr 13, 2020

joshgoebel Apr 14, 2020

pplantinga Apr 14, 2020

joshgoebel Apr 14, 2020

joshgoebel Apr 14, 2020

pplantinga Apr 14, 2020

joshgoebel Apr 14, 2020

pplantinga Apr 14, 2020

pplantinga Apr 14, 2020

joshgoebel Apr 14, 2020

joshgoebel Apr 14, 2020

pplantinga Apr 14, 2020

pplantinga Apr 14, 2020

joshgoebel Apr 17, 2020

joshgoebel Apr 17, 2020 •

edited

pplantinga Apr 17, 2020 •

edited

joshgoebel commented Apr 17, 2020

pplantinga commented Apr 17, 2020

joshgoebel commented Apr 17, 2020

joshgoebel commented Apr 17, 2020

pplantinga commented Apr 17, 2020

joshgoebel commented Apr 21, 2020

(yaml) improve tag support; add verbatim tags #2487

(yaml) improve tag support; add verbatim tags #2487

Conversation

pplantinga commented Apr 9, 2020 • edited by joshgoebel

joshgoebel left a comment

Choose a reason for hiding this comment

joshgoebel commented Apr 9, 2020

pplantinga commented Apr 10, 2020

joshgoebel commented Apr 10, 2020

pplantinga commented Apr 12, 2020

joshgoebel commented Apr 13, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joshgoebel Apr 17, 2020 • edited

Choose a reason for hiding this comment

pplantinga Apr 17, 2020 • edited

Choose a reason for hiding this comment

joshgoebel commented Apr 17, 2020

pplantinga commented Apr 17, 2020

joshgoebel commented Apr 17, 2020

joshgoebel commented Apr 17, 2020

pplantinga commented Apr 17, 2020

joshgoebel commented Apr 21, 2020

pplantinga commented Apr 9, 2020 •

edited by joshgoebel

joshgoebel Apr 17, 2020 •

edited

pplantinga Apr 17, 2020 •

edited