Can't add additional custom tokenizer or renderers?

[alystair]

What pain point are you perceiving?.
I'm reviewing Marked documentation, attempting to create a custom setup where, it transforms new lines starting with 'notice: ' into a specifically formatted DIV. By my understanding I need to first add a custom named tokenizer and then a renderer based on it? Or am I going about this the wrong way?

Describe the solution you'd like
I'd like to easily add this functionality without creating a separate post-processing tool external to Marked...

[KilianKilmister]

You are propably able to get by creating a walk-token. They basically act as an intermediate processing step before the data is handed to the renderer. Their ability is somewhat limited, but it should be plenty for things like this.

[alystair]

I tried the following but it doesn't work. Does tokenizer only substitute existing tokens? Can you not add your own custom ones?

const tokenizer = {
	tipNotice(src) {
		const match = src.match(/\nnotice: (.*)\n/);
		if (match) {
			return {
				type:'tipNotice',
				raw:match[0],
				text:match[1].trim()
			};
		}
		return false;
	}
};
const renderer = {
	tipNotice(text) {
		return `<div class="tip notice">${text}</div>`;
	}
};
marked.use({ tokenizer, renderer });
marked(content,{ headerIds:true });

[KilianKilmister]

@alystair Both the Parser and the Lexer implementation currently have their structure hardcoded. So you can so there currently is no straight way of altering what they hand to the Tokenizer and Renderer, as those are "stupid" as they are build tor speedy processing.

And as it turnes out, it is pretty much impossible to make invasive modifications short of altering the source-code or meta programming a weil.

Whoever wrote the core of this package can really pad himself on the back, it is as solid as granit.

#1695 i filed an issue about that yesterday

[alystair]

Argh... in that case I'll have to replace several built-ins like paragraph, and review the source code to expand on the current functionality. Fun.

[KilianKilmister]

A little hint: copy the repo and directly modify the sourcecode i spent a full day trying to alter the parser and lexer from inside my module and i have nothing to show for it, as marked is stable as hell.

But the code is modern and well formatted. Navigating it is a breze

[UziTech]

This is something that could be useful. If someone wants to create a PR I would be happy to review it.

[thien-do]

@UziTech the PR is to add "token" param to renderer methods, right? So for example the "paragraph" one will receive "token" in addition to "text"?

[UziTech]

No the issue is for the ability to add additional tokenizers/renderers. For example adding an "underline" tokenizers and renderer to extend markdown.

[UziTech]

Moving to the renderer taking tokens instead of certain parameters is something that will have to be done on a major version bump but I think it would help with this issue.

[alystair]

Any chance of this being done internally or you'll definitely be waiting for a 3rd party PR?

[UziTech]

There is already a PR to add the ability to extend the tokenizer #1872 but it hasn't been worked on in a while.

[calculuschild]

We kind of got stuck on #1872 due to some slowdown issues, and other things had higher priority at the time so it got left behind. There was discussion of a possible different approach that might work better, but I'm not sure if that's still the route we want to end up taking. @UziTech Is that still our preferred approach? Perhaps it would be worth setting up a discussion page to determine how exactly we want extensions to work in the future with a clear structure before revisiting that PR.

[UziTech]

@calculuschild I think we can continue the conversation on that PR. The biggest thing is keeping marked fast for users that don't use that feature.

@alystair If you wanted to create a PR with a solution it would definitely help.

[dawgdemi]

cd utils
node build.js

[UziTech]

This Should be available in v2.1.0

see Custom Extensions section in docs