HTML API: Roadmap

[dmsnell]

HTML issues | Refactors | Interesting Patches | Broader Roadmap | Plans for 6.6 | Plans for 6.7

⚠️ Note: This planning document has been abandoned in favor of the HTML API: Roadmap issue.

Untriaged plans.

• Figure out and expose whitespace collapsing rules so that it's possible to make proper HTML-to-text functions without recreating that logic in consuming code.
  • Unfortunately it looks like this is mostly handled through the styling engine. We can pre-process a document to collapse \r\n into \n and then swap \r for \n, but beyond that, there aren't specific rules for inter-element whitespace on render. Render is governed by a complicated interaction between elements.
  • For instance, we should create a newline for paragraphs, but not if they are the first text content inside of an LI. We could have <li><p>Stuff and this should only have a single newline.
• Parsing rules change in SVG and MathML content. This needs to be understood by both the Tag Processor and the HTML Processor.
• Add DOM-like methods: set_inner_html(), get_inner_html(), wrap_with(), unwrap(), etc…
  • WP_HTML_Processor for replacing content inside tag #59623
  • Add method to get inner/outer HTML in WP_HTML_Tag_Processor #60046
• Provide a listener interface for when popping elements off of the stack of open elements.
  • This is necessary to enable a number of behaviors related to identifying an open tag and where it ends.
• Provide ability to extend the current document with new chunks of HTML.
  • A retention mode preserves the existing HTML so that get_updated_html() returns the full document with all chunks. extend( string $next_chunk ): ? makes the internal HTML document bigger and need not return anything.
  • A forgetful mode releases the existing HTML so that get_updated_html() will only be able to return the contents of the next chunk. chunk_slide( string $next_chunk ): string will extend the document, but will also release as much of the previous document that it can, returning the fully-updated portion of the total HTML document that is no longer reachable by a bookmark.
• Replace html_decode_entities() with a version that follows HTML's rules, particularly surrounding the ; use and the ambiguous ampersand rule.
• Add memory and runtime limits for arbitrarily constrained environments.
  • A memory limit essentially sets a maximum chunk size and forces forgetful streaming mode, but will operate in less memory than is required to load the entire HTML string into memory. Useful for streaming pipelines where it's not required to contain the entire document at once.
  • A runtime limit can halt processing after a given timeout and disable further operations other than get_updated_html().
• Create a read-only copy of the Tag/HTML Processor that's safe to pass into filters and functions so that they can read from the document without changing it.
• Provide the ability to limit edits to a specified region of a document; allowing some operations, e.g. let a filter modify the attributes on the current tag only.
• Communicate in the HTML Processor that the semantic rules for the current token imply the creation of DOM nodes to a prior location in the document (active format reconstruction, adoption, and fostering, duplicate HTML and BODY tags).
  • An unexpected </p> implicitly creates a <p> to form an empty P element, but the HTML processor will not find the opening P tag because it doesn't exist. A CSS selector for P might miss this or any node it's targeting that depends on nth-child semantics.
  • Active format reconstruction creates new formatting elements, sometimes at a reasonable distance before the current tag. The HTML processor will not find these tags because they don't exist.
  • A DIV element as an apparent direct child of a TABLE will be shifted to become a previous sibling of the TABLE element. The HTML processor will not have seen the DIV before the TABLE, even though a DOM would.
  • A duplicate BODY tag sets attributes on the BODY element which don't already exist. The HTML processor would mistakenly report the BODY attributes because it didn't know more were to come.
  • These situations may or may not present obstacles for processing code, dependent on the context. E.g. if it's not relevant that the BODY inherits new attributes, the processor need not halt. In some situations it will be essential: e.g. "replace all empty paragraphs with some content" needs to find that implicit empty P element; "add an attribute to every B element" needs to find the implicit B tag that is created later during reconstruction.

Tasks

Bug fixes and quality

• We need to defer applying enqueued edits as much as we can. When we removed that optimization in order to make the code simpler, we overlooked that on documents with many edits, this could lead to cataclysmic runtime overhead both in processing and memory. The workaround is to track edits externally and apply them all at once. The mechanism is that when applying an edit we copy the entire document, so if we have 50 edits we copy the entire document 50 times. With deferred updates, we only copy the entire document once when applying all of them in one go. [HTML API: Defer applying updates until necessary. wordpress-develop#6120]

Waiting review and merge

Future releases

• Provide new filtering pipelines for final rendered HTML. Core-43258

WordPress 6.6

• Safe get/set inner HTML
• HTML Templating for safe HTML generation.
• New render-pipeline filter replacing Core functionality:
  • smilies/emojify
  • capital_P_dangit
• Core refactors for HTML processing
  • wp_strip_all_tags()
  • force_balance_tags() rewritten as "serialize this HTML"

WordPress 6.5

• Support in the HTML Processor for most common tags IN BODY.
• Scan all tokens in the Tag Processor to enable modifying HTML structure.
• HTML Templating for safe HTML generation.
• Establish test suite of real posts and websites against which to run the HTML Processor and report progress.
• Run the html5lib tests against the HTML Processor in WordPress CI suite.

WordPress 6.4

Merged and bound for 6.4

• HTML API: Stop proceeding HTML when encountering unsupported markup. wordpress-develop#5048
• HTML API: Update documentation and rename internal variable on HTML Processor wordpress-develop#5126
• HTML API: Store current token reference in HTML Processor state. wordpress-develop#5127
• HTML API: Skip over contents of RAWTEXT elements such as STYLE. wordpress-develop#5145
• Block Supports: Backport update from Gutenberg wordpress-develop#5252
• HTML API: Remove all duplicate copies of an attribute when removing wordpress-develop#4317
• HTML API: Add class name utilities has_class() and class_list() wordpress-develop#5096
• HTML API: Add matches_breadcrumbs() to HTML Processor for better querying wordpress-develop#5243

In progress for WordPress 6.4

• Support for elements in IN BODY mode.
• HTML API: Only pass a single class name to add_class() wordpress-develop#5325
  • this probably won't merge, but parts of it might go out. we want to remove code that's passing a plurality of things to a singularity of thing. the issue might seem pedantic, but this can cause defects in class handling and we need to find the right way to resolve it.

Plans for post-6.4 merges

• Need to refactor the Tag Processor to think more about "tokens" than "tags" internally so that we can stop on comments and other non-tag tokens. This will not only support "funky comments" but is also necessary for work like the wp_strip_tags() and truncate_html() functionality, which needs to read plaintext content of markup (which needs to ignore comment and other meta content).
• Focus on adding HTML templating so that the HTML API becomes useful for safe HTML generation in all the places we're currently forgetting to escape attributes and the like.

PRs to revisit

• Search block: refactor to use HTML Tag Processor #51273 - see if we can unwind the unsafe concatenation of HTML inside the constructor
• behaviors/lightbox - see if we can remove using multiple instances

Areas of active exploration

• Expose "original raw tag" for backwards compatibility with filters that expect spans of the HTML document from functions such as wp_kses_hair(). [HTML API: Expose raw tag markup to support existing filters wordpress-develop#5143]
  • This is probably best left for an internal Core class, which is also needed for several of Core's cleanup tasks, tasks that need to examine raw markup and avoid making needless changes.
• Add set_raw_inner_markup() and get_raw_inner_markup() (or not, if it's not the right interface). [HTML API: add get/set inner/outer markup wordpress-develop#4956]
• A new wp_strip_tags() function/approach that only parses as much HTML as is necessary. [WIP: HTML API: Extract previous text and HTML chunks while processing. wordpress-develop#5208]
• Allow extending the input document for more strict streaming work. [WIP: HTML API: Allow extending input document for chunked processing. wordpress-develop#5050]

HTML Templating

Provide a means to generate HTML conveniently with placeholders. The placeholders should be "funky comments" that mirror array values passed in to the rendering function. This will/should form the basis for raw HTML templating, replacing inner contents, powering Bits so that we can apply heuristics to the replacement markup, and more.