Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

use correct options in specs #1511

Merged
merged 13 commits into from Jul 2, 2019
Merged

use correct options in specs #1511

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
4b6a6e7
.md files can have skip and only to help debug
UziTech Jul 1, 2019
ef4904d
use correct options
UziTech Jul 1, 2019
900114a
move fences to normal
UziTech Jul 1, 2019
57f6d89
improve heading
UziTech Jul 1, 2019
ba2eedd
improve lheading
UziTech Jul 1, 2019
8ccc1f1
improve paragraph
UziTech Jul 1, 2019
f478dfb
move tables to gfm
UziTech Jul 1, 2019
7bc6e3f
add comment
UziTech Jul 1, 2019
7ed2692
fix testing with node 4
UziTech Jul 1, 2019
d2e85d9
fix fences
UziTech Jul 2, 2019
44dbeeb
remove tables option
UziTech Jul 2, 2019
1f4c17f
Merge branch 'master' into spec-options
UziTech Jul 2, 2019
0276ad7
silent tests
UziTech Jul 2, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 0 additions & 2 deletions docs/USING_ADVANCED.md
View file
Open in desktop
Expand Up @@ -25,7 +25,6 @@ marked.setOptions({
},
pedantic: false,
gfm: true,
tables: true,
styfle marked this conversation as resolved.
Show resolved Hide resolved
breaks: false,
sanitize: false,
smartLists: true,
Expand Down Expand Up @@ -56,7 +55,6 @@ console.log(marked(markdownString));
|silent |`boolean` |`false` |v0.2.7 |If true, the parser does not throw any exception.|
|smartLists |`boolean` |`false` |v0.2.8 |If true, use smarter list behavior than those found in `markdown.pl`.|
|smartypants |`boolean` |`false` |v0.2.9 |If true, use "smart" typographic punctuation for things like quotes and dashes.|
|tables |`boolean` |`true` |v0.2.7 |If true and `gfm` is true, use [GFM Tables extension](https://github.github.com/gfm/#tables-extension-).|
|xhtml |`boolean` |`false` |v0.3.2 |If true, emit self-closing HTML tags for void elements (<br/>, <img/>, etc.) with a "/" as required by XHTML.|

<h2 id="highlight">Asynchronous highlighting</h2>
Expand Down
64 changes: 30 additions & 34 deletions lib/marked.js
View file
Open in desktop
Expand Up @@ -14,10 +14,9 @@
var block = {
newline: /^\n+/,
code: /^( {4}[^\n]+\n*)+/,
fences: noop,
fences: /^ {0,3}(`{3,}|~{3,})([^`~\n]*)\n(?:|([\s\S]*?)\n)(?: {0,3}\1[~`]* *(?:\n+|$)|$)/,
hr: /^ {0,3}((?:- *){3,}|(?:_ *){3,}|(?:\* *){3,})(?:\n+|$)/,
heading: /^ *(#{1,6}) *([^\n]+?) *(?:#+ *)?(?:\n+|$)/,
nptable: noop,
heading: /^ {0,3}(#{1,6}) +([^\n]*?)(?: +#+)? *(?:\n+|$)/,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks OK

blockquote: /^( {0,3}> ?(paragraph|[^\n]*)(?:\n|$))+/,
list: /^( {0,3})(bull) [\s\S]+?(?:hr|def|\n{2,}(?! )(?!\1bull )\n*|\s*$)/,
html: '^ {0,3}(?:' // optional indentation
Expand All @@ -31,9 +30,12 @@ var block = {
+ '|</(?!script|pre|style)[a-z][\\w-]*\\s*>(?=[ \\t]*(?:\\n|$))[\\s\\S]*?(?:\\n{2,}|$)' // (7) closing tag
+ ')',
def: /^ {0,3}\[(label)\]: *\n? *<?([^\s>]+)>?(?:(?: +\n? *| *\n *)(title))? *(?:\n+|$)/,
nptable: noop,
table: noop,
lheading: /^([^\n]+)\n {0,3}(=|-){2,} *(?:\n+|$)/,
paragraph: /^([^\n]+(?:\n(?!hr|heading|lheading| {0,3}>|<\/?(?:tag)(?: +|\n|\/?>)|<(?:script|pre|style|!--))[^\n]+)*)/,
lheading: /^([^\n]+)\n {0,3}(=+|-+) *(?:\n+|$)/,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK

// regex template, placeholders will be replaced according to different paragraph
// interruption rules of commonmark and the original markdown spec:
_paragraph: /^([^\n]+(?:\n(?!hr|heading|lheading|blockquote|fences|list|html)[^\n]+)*)/,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK

text: /^[^\n]+/
};

Expand Down Expand Up @@ -69,10 +71,14 @@ block.html = edit(block.html, 'i')
.replace('attribute', / +[a-zA-Z:_][\w.:-]*(?: *= *"[^"\n]*"| *= *'[^'\n]*'| *= *[^\s"'=<>`]+)?/)
.getRegex();

block.paragraph = edit(block.paragraph)
block.paragraph = edit(block._paragraph)
.replace('hr', block.hr)
.replace('heading', block.heading)
.replace('lheading', block.lheading)
.replace('heading', ' {0,3}#{1,6} +')
.replace('|lheading', '') // setex headings don't interrupt commonmark paragraphs
.replace('blockquote', ' {0,3}>')
.replace('fences', ' {0,3}(?:`{3,}|~{3,})[^`\\n]*\\n')
.replace('list', ' {0,3}(?:[*+-]|1[.)]) ') // only lists starting from 1 can interrupt
.replace('html', '</?(?:tag)(?: +|\\n|/?>)|<(?:script|pre|style|!--)')
.replace('tag', block._tag) // pars can be interrupted by type (6) html blocks
.getRegex();

Expand All @@ -91,28 +97,12 @@ block.normal = merge({}, block);
*/

block.gfm = merge({}, block.normal, {
fences: /^ {0,3}(`{3,}|~{3,})([^`\n]*)\n(?:|([\s\S]*?)\n)(?: {0,3}\1[~`]* *(?:\n+|$)|$)/,
paragraph: /^/,
heading: /^ *(#{1,6}) +([^\n]+?) *#* *(?:\n+|$)/
});

block.gfm.paragraph = edit(block.paragraph)
.replace('(?!', '(?!'
+ block.gfm.fences.source.replace('\\1', '\\2') + '|'
+ block.list.source.replace('\\1', '\\3') + '|')
.getRegex();

/**
* GFM + Tables Block Grammar
*/

block.tables = merge({}, block.gfm, {
nptable: /^ *([^|\n ].*\|.*)\n *([-:]+ *\|[-| :]*)(?:\n((?:.*[^>\n ].*(?:\n|$))*)\n*|$)/,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Super-linear.

  1. The construct .*\|.* can be exploited through a run of |||...|. Can we replace with [^|]*|[^|]* ?
  2. I believe the same problem applies later in the (?:.*[^>\n ].* section.

table: /^ *\|(.+)\n *\|?( *[-:]+[-| :]*)(?:\n((?: *[^>\n ].*(?:\n|$))*)\n*|$)/
Copy link
Contributor

@davisjam davisjam Jul 2, 2019 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same genre of problem as the nptable.

if (/^ *\|(.+)\n *\|?( *[-:]+[-| :]*)(?:\n((?: *[^>\n ].*(?:\n|$))*)\n*|$)/.exec(' | \n' + ' '.repeat(50000))) {
  console.log('match');
}

(Note, here and elsewhere I'm just checking regexes, not full exploitability).

});

/**
* Pedantic grammar
* Pedantic grammar (original John Gruber's loose markdown specification)
*/

block.pedantic = merge({}, block.normal, {
Expand All @@ -126,7 +116,18 @@ block.pedantic = merge({}, block.normal, {
+ '|sup|i|b|u|mark|ruby|rt|rp|bdi|bdo|span|br|wbr|ins|del|img)'
+ '\\b)\\w+(?!:|[^\\w\\s@]*@)\\b')
.getRegex(),
def: /^ *\[([^\]]+)\]: *<?([^\s>]+)>?(?: +(["(][^\n]+[")]))? *(?:\n+|$)/
def: /^ *\[([^\]]+)\]: *<?([^\s>]+)>?(?: +(["(][^\n]+[")]))? *(?:\n+|$)/,
heading: /^ *(#{1,6}) *([^\n]+?) *(?:#+ *)?(?:\n+|$)/,
fences: noop, // fences not supported
paragraph: edit(block.normal._paragraph)
.replace('hr', block.hr)
.replace('heading', ' *#{1,6} *[^\n]')
.replace('lheading', block.lheading)
.replace('blockquote', ' {0,3}>')
.replace('|fences', '')
.replace('|list', '')
.replace('|html', '')
.getRegex()
});

/**
Expand All @@ -142,11 +143,7 @@ function Lexer(options) {
if (this.options.pedantic) {
this.rules = block.pedantic;
} else if (this.options.gfm) {
if (this.options.tables) {
this.rules = block.tables;
} else {
this.rules = block.gfm;
}
this.rules = block.gfm;
}
}

Expand Down Expand Up @@ -233,7 +230,7 @@ Lexer.prototype.token = function(src, top) {
continue;
}

// fences (gfm)
// fences
if (cap = this.rules.fences.exec(src)) {
src = src.substring(cap[0].length);
this.tokens.push({
Expand Down Expand Up @@ -494,7 +491,7 @@ Lexer.prototype.token = function(src, top) {
src = src.substring(cap[0].length);
this.tokens.push({
type: 'heading',
depth: cap[2] === '=' ? 1 : 2,
depth: cap[2].charAt(0) === '=' ? 1 : 2,
text: cap[1]
});
continue;
Expand Down Expand Up @@ -1663,7 +1660,6 @@ marked.getDefaults = function() {
silent: false,
smartLists: false,
smartypants: false,
tables: true,
xhtml: false
};
};
Expand Down
5 changes: 1 addition & 4 deletions man/marked.1
View file
Open in desktop
Expand Up @@ -8,7 +8,7 @@ marked \- a javascript markdown parser
.B marked
[\-o \fI<output>\fP] [\-i \fI<input>\fP] [\-\-help]
[\-\-tokens] [\-\-pedantic] [\-\-gfm]
[\-\-breaks] [\-\-tables] [\-\-sanitize]
[\-\-breaks] [\-\-sanitize]
[\-\-smart\-lists] [\-\-lang\-prefix \fI<prefix>\fP]
[\-\-no\-etc...] [\-\-silent] [\fIfilename\fP]

Expand Down Expand Up @@ -72,9 +72,6 @@ Enable github flavored markdown.
.BI \-\-breaks
Enable GFM line breaks. Only works with the gfm option.
.TP
.BI \-\-tables
Enable GFM tables. Only works with the gfm option.
.TP
.BI \-\-sanitize
Sanitize output. Ignore any HTML input.
.TP
Expand Down
11 changes: 4 additions & 7 deletions man/marked.1.txt
View file
Open in desktop
Expand Up @@ -4,10 +4,10 @@ NAME
marked - a javascript markdown parser

SYNOPSIS
marked [-o <output>] [-i <input>] [--help] [--tokens]
[--pedantic] [--gfm] [--breaks] [--tables] [--sanitize]
[--smart-lists] [--lang-prefix <prefix>] [--no-etc...] [--silent]
[filename]
marked [-o <output>] [-i <input>] [--help] [--tokens] [--pedantic]
[--gfm] [--breaks] [--sanitize] [--smart-lists] [--lang-prefix <pre-
fix>] [--no-etc...] [--silent] [filename]


DESCRIPTION
marked is a full-featured javascript markdown parser, built for speed.
Expand Down Expand Up @@ -56,9 +56,6 @@ OPTIONS
--breaks
Enable GFM line breaks. Only works with the gfm option.

--tables
Enable GFM tables. Only works with the gfm option.

--sanitize
Sanitize output. Ignore any HTML input.

Expand Down
3 changes: 0 additions & 3 deletions test/bench.js
View file
Open in desktop
Expand Up @@ -29,7 +29,6 @@ function runBench(options) {
// Non-GFM, Non-pedantic
marked.setOptions({
gfm: false,
tables: false,
breaks: false,
pedantic: false,
sanitize: false,
Expand All @@ -43,7 +42,6 @@ function runBench(options) {
// GFM
marked.setOptions({
gfm: true,
tables: false,
breaks: false,
pedantic: false,
sanitize: false,
Expand All @@ -57,7 +55,6 @@ function runBench(options) {
// Pedantic
marked.setOptions({
gfm: false,
tables: false,
breaks: false,
pedantic: true,
sanitize: false,
Expand Down
8 changes: 7 additions & 1 deletion test/helpers/load.js
View file
Open in desktop
Expand Up @@ -79,11 +79,17 @@ function loadFiles(dir) {
switch (ext) {
case '.md': {
const content = fm(fs.readFileSync(absFile, 'utf8'));
const skip = content.attributes.skip;
delete content.attributes.skip;
const only = content.attributes.only;
delete content.attributes.only;
specs = [{
section: name,
markdown: content.body,
html: fs.readFileSync(absFile.replace(/[^.]+$/, 'html'), 'utf8'),
options: content.attributes
options: content.attributes,
only,
skip
}];
break;
}
Expand Down
33 changes: 11 additions & 22 deletions test/specs/commonmark/commonmark.0.29.json
View file
Open in desktop
Expand Up @@ -358,17 +358,15 @@
"example": 45,
"start_line": 915,
"end_line": 919,
"section": "ATX headings",
"shouldFail": true
"section": "ATX headings"
},
{
"markdown": "### foo \\###\n## foo #\\##\n# foo \\#\n",
"html": "<h3>foo ###</h3>\n<h2>foo ###</h2>\n<h1>foo #</h1>\n",
"example": 46,
"start_line": 925,
"end_line": 933,
"section": "ATX headings",
"shouldFail": true
"section": "ATX headings"
},
{
"markdown": "****\n## foo\n****\n",
Expand Down Expand Up @@ -427,8 +425,7 @@
"example": 53,
"start_line": 1046,
"end_line": 1055,
"section": "Setext headings",
"shouldFail": true
"section": "Setext headings"
},
{
"markdown": " Foo\n---\n\n Foo\n-----\n\n Foo\n ===\n",
Expand Down Expand Up @@ -508,8 +505,7 @@
"example": 63,
"start_line": 1186,
"end_line": 1196,
"section": "Setext headings",
"shouldFail": true
"section": "Setext headings"
},
{
"markdown": "- Foo\n---\n",
Expand Down Expand Up @@ -1650,8 +1646,7 @@
"example": 205,
"start_line": 3464,
"end_line": 3476,
"section": "Block quotes",
"shouldFail": true
"section": "Block quotes"
},
{
"markdown": "> foo\n bar\n",
Expand Down Expand Up @@ -2220,8 +2215,7 @@
"example": 274,
"start_line": 5005,
"end_line": 5011,
"section": "Lists",
"shouldFail": true
"section": "Lists"
},
{
"markdown": "The number of windows in my house is\n1. The number of doors is 6.\n",
Expand Down Expand Up @@ -2366,8 +2360,7 @@
"example": 291,
"start_line": 5379,
"end_line": 5397,
"section": "Lists",
"shouldFail": true
"section": "Lists"
},
{
"markdown": "- a\n",
Expand Down Expand Up @@ -4904,8 +4897,7 @@
"example": 598,
"start_line": 8787,
"end_line": 8791,
"section": "Autolinks",
"shouldFail": true
"section": "Autolinks"
},
{
"markdown": "<http://example.com/\\[\\>\n",
Expand Down Expand Up @@ -4953,8 +4945,7 @@
"example": 604,
"start_line": 8850,
"end_line": 8854,
"section": "Autolinks",
"shouldFail": true
"section": "Autolinks"
},
{
"markdown": "<m:abc>\n",
Expand All @@ -4978,17 +4969,15 @@
"example": 607,
"start_line": 8871,
"end_line": 8875,
"section": "Autolinks",
"shouldFail": true
"section": "Autolinks"
},
{
"markdown": "foo@bar.example.com\n",
"html": "<p>foo@bar.example.com</p>\n",
"example": 608,
"start_line": 8878,
"end_line": 8882,
"section": "Autolinks",
"shouldFail": true
"section": "Autolinks"
},
{
"markdown": "<a><bab><c2c>\n",
Expand Down