Nokogumbo provides the ability for a Ruby program to invoke the Gumbo HTML5 parser and to access the result as a Nokogiri::HTML::Document.
require 'nokogumbo'
doc = Nokogiri.HTML5(string)
An experimental fragment method is also provided. While not HTML5 compliant, it may be useful:
require 'nokogumbo'
doc = Nokogiri::HTML5.fragment(string)
Because HTML is often fetched via the web, a convenience interface to HTTP get is also provided:
require 'nokogumbo'
doc = Nokogiri::HTML5.get(uri)
The document and fragment parsing methods,
Nokogiri.HTML5(html, url = nil, encoding = nil, options = {})
Nokogiri::HTML5.parse(html, url = nil, encoding = nil, options = {})
Nokogiri::HTML5::Document.parse(html, url = nil, encoding = nil, options = {})
Nokogiri::HTML5.fragment(html, encoding = nil, options = {})
Nokogiri::HTML5::DocumentFragment.parse(html, encoding = nil, options = {})
support options that are different from Nokogiri's.
The two currently supported options are :max_errors
and :max_tree_depth
,
described below.
Nokogumbo contains an experimental parse error reporting facility. By default,
no parse errors are reported but this can be configured by passing the
:max_errors
option to ::parse
or ::fragment
.
require 'nokogumbo'
doc = Nokogiri::HTML5.parse('Hi there!<body>', max_errors: 10)
doc.errors.each do |err|
puts err
end
This prints the following.
1:1: ERROR: @1:1: The doctype must be the first token in the document.
Hi there!<body>
^
1:10: ERROR: @1:10: That tag isn't allowed here Currently open tags: html, body..
Hi there!<body>
^
Using max_errors: -1
results in an unlimited number of errors being
returned.
The errors returned by #errors
are instances of
Nokogiri::XML::SyntaxError
.
The maximum depth of the DOM tree parsed by the various parsing methods is
configurable by the :max_tree_depth
option. If the depth of the tree would
exceed this limit, then an
ArgumentError is thrown.
This limit (which defaults to Nokogumbo::DEFAULT_MAX_TREE_DEPTH = 400
) can
be removed by giving the option max_tree_depth: -1
.
html = '<!DOCTYPE html>' + '<div>' * 1000
doc = Nokogiri.HTML5(html)
# raises ArgumentError: Document tree depth limit exceeded
doc = Nokogiri.HTML5(html, max_tree_depth: -1)
After parsing HTML, it may be serialized using any of the Nokogiri
serialization
methods. In
particular, #serialize
, #to_html
, and #to_s
will serialize a given node
and its children. (This is the equivalent of JavaScript's
Element.outerHTML
.) Similarly, #inner_html
will serialize the children of
a given node. (This is the equivalent of JavaScript's Element.innerHTML
.)
doc = Nokogiri::HTML5("<!DOCTYPE html><span>Hello world!</span>")
puts doc.serialize
# Prints: <!DOCTYPE html><html><head></head><body><span>Hello world!</span></body></html>
Due to quirks in how HTML is parsed and serialized, it's possible for a DOM tree to be serialized and then re-parsed, resulting in a different DOM. Mostly, this happens with DOMs produced from invalid HTML. Unfortunately, even valid HTML may not survive serialization and re-parsing.
In particular, a newline at the start of pre
, listing
, and textarea
elements is ignored by the parser.
doc = Nokogiri::HTML5(<<-EOF)
<!DOCTYPE html>
<pre>
Content</pre>
EOF
puts doc.at('/html/body/pre').serialize
# Prints: <pre>Content</pre>
In this case, the original HTML is semantically equivalent to the serialized
version. If the pre
, listing
, or textarea
content starts with two
newlines, the first newline will be stripped on the first parse and the second
newline will be stripped on the second, leading to semantically different
DOMs. Passing the parameter preserve_newline: true
will cause two or more
newlines to be preserved. (A single leading newline will still be removed.)
doc = Nokogiri::HTML5(<<-EOF)
<!DOCTYPE html>
<listing>
Content</listing>
EOF
puts doc.at('/html/body/listing').serialize(preserve_newline: true)
# Prints: <listing>
#
# Content</listing>
Nokogumbo always parses HTML using
UTF-8; however, the encoding of the
input can be explicitly selected via the optional encoding
parameter. This
is most useful when the input comes not from a string but from an IO object.
When serializing a document or node, the encoding of the output string can be
specified via the :encoding
options. Characters that cannot be encoded in
the selected encoding will be encoded as HTML numeric
entities.
frag = Nokogiri::HTML5.fragment('<span>아는 길도 물어가라</span>')
html = frag.serialize(encoding: 'US-ASCII')
puts html
# Prints: <span>아는 길도 물어가라</span>
frag = Nokogiri::HTML5.fragment(html)
puts frag.serialize
# Prints: <span>아는 길도 물어가라</span>
(There's a bug in all current
versions of Ruby that can cause the entity encoding to fail. Of the mandated
supported encodings for HTML, the only encoding I'm aware of that has this bug
is 'ISO-2022-JP'
. I recommend avoiding this encoding.)
require 'nokogumbo'
puts Nokogiri::HTML5.get('http://nokogiri.org').search('ol li')[2].text
-
The
Nokogiri::HTML5.fragment
function takes a string and parses it as a HTML5 document. The<html>
,<head>
, and<body>
elements are removed from this document, and any children of these elements that remain are returned as aNokogiri::HTML::DocumentFragment
. -
The
Nokogiri::HTML5.parse
function takes a string and passes it to thegumbo_parse_with_options
method, using the default options. The resulting Gumbo parse tree is then walked.- If the necessary Nokogiri and libxml2 headers can be found at installation time then an xmlDoc tree is produced and a single Nokogiri Ruby object is constructed to wrap the xmlDoc structure. Nokogiri only produces Ruby objects as necessary, so all searching is done using the underlying libxml2 libraries.
- If the necessary headers are not present at installation time, then Nokogiri Ruby objects are created for each Gumbo node. Other than memory usage and CPU time, the results should be equivalent.
-
The
Nokogiri::HTML5.get
function takes care of following redirects, https, and determining the character encoding of the result, based on the rules defined in the HTML5 specification for doing so. -
Instead of uppercase element names, lowercase element names are produced.
-
Instead of returning
unknown
as the element name for unknown tags, the original tag name is returned verbatim.
Nokogumbo uses libxml2, the XML library underlying Nokogiri, to speed up parsing. If the libxml2 headers are not available, then Nokogumbo resorts to using Nokogiri's Ruby API to construct the DOM tree.
Nokogiri can be configured to either use the system library version of libxml2 or use a bundled version. By default (as of Nokogiri version 1.8.4), Nokogiri will use a bundled version.
To prevent differences between versions of libxml2, Nokogumbo will only use libxml2 if the build process can find the exact same version used by Nokogiri. This leads to three possibilities
- Nokogiri is compiled with the bundled libxml2. In this case, Nokogumbo will (by default) use the same version of libxml2.
- Nokogiri is compiled with the system libxml2. In this case, if the libxml2 headers are available, then Nokogumbo will (by default) use the system version and headers.
- Nokogiri is compiled with the system libxml2 but its headers aren't available at build time for Nokogumbo. In this case, Nokogumbo will use the slower Ruby API.
Using libxml2 can be required by passing -- --with-libxml2
to bundle exec rake
or to gem install
. Using libxml2 can be prohibited by instead passing
-- --without-libxml2
.
Functionally, the only difference between using libxml2 or not is in the
behavior of Nokogiri::XML::Node#line
. If it is used, then #line
will
return the line number of the corresponding node. Otherwise, it will return 0.
git clone https://github.com/rubys/nokogumbo.git
cd nokogumbo
bundle install
rake gem
gem install pkg/nokogumbo*.gem
- ruby-gumbo -- a ruby binding for the Gumbo HTML5 parser.
- lua-gumbo -- a lua binding for the Gumbo HTML5 parser.