Documents and Fragments

Before you can scrub anything you must parse it, and Loofah gives you six ways to do that: HTML5, HTML4, or XML, each as a full document or as a fragment. This page explains how to choose, what each class adds on top of Nokogiri, and the parsing details that affect the result.

Fragments vs Documents

Unless you expect a DOCTYPE and a single root element, you have a fragment, not a document. For HTML, a useful rule of thumb is that documents have <html> and <body> tags and fragments do not.

Fragment Document
Typical input Comment body, rich-text editor output, an email snippet A complete web page
Wrapped in html/body? No Yes, the parser adds them if missing
Multiple top-level nodes Allowed One root element
to_s returns Only the fragment's markup The whole document, including <html>, <head>, <body>
text reads from The fragment's top-level nodes /html/body only

For HTML fragments, the README notes that head elements are silently ignored by the parser.

Loofah.html5_fragment("<p>one</p><p>two</p>").to_s
# => "<p>one</p><p>two</p>"

Loofah.html5_document("<p>one</p><p>two</p>").to_s
# => output similar to "<html><head></head><body><p>one</p><p>two</p></body></html>"

Tip: If you parse a snippet as a document and then serialize the whole document, you will get <html> and <body> wrappers in your output. Use a fragment, or serialize only doc.at_css("body").inner_html.

HTML4 vs HTML5

Loofah can use either of Nokogiri's HTML parsers:

HTML4 (html4_*) HTML5 (html5_*)
Parser libxml2 (CRuby) or NekoHTML (JRuby) Gumbo, a spec-compliant HTML5 parser
Availability Everywhere CRuby with Nokogiri newer than 1.14.0 (details)
Matches browser behavior Approximately Follows the WHATWG parsing algorithm
Default for Loofah.document / Loofah.fragment Yes No

The project strongly recommends calling html5_document / html5_fragment explicitly unless you have a compelling reason not to. The reason is security as much as correctness: the browser that eventually renders your sanitized HTML uses an HTML5 parser, so sanitizing with the same parsing rules removes a whole class of "parser mismatch" attacks. The repository's design note on CDATA nodes describes several past vulnerabilities in this class and concludes that HTML5 parsing makes it go away.

If you must use HTML4, call html4_document / html4_fragment explicitly. The generic aliases Loofah.document, Loofah.fragment, Loofah.scrub_document, and Loofah.scrub_fragment currently point to the HTML4 methods, and the README warns that they may switch to HTML5 in a future version. Likewise Loofah::HTML is an alias for Loofah::HTML4.

Visible differences between the two parsers include:

  • An empty string parsed as a document: HTML5 produces <html><head></head><body></body></html>, HTML4 produces an empty document with no root (doc.root is nil).
  • Attribute serialization: HTML4 escapes < inside attribute values (alt="this &lt; that"), HTML5 leaves it as alt="this < that".
  • Serializing a document with to_s: on CRuby the HTML4 document includes a <!DOCTYPE ...> line, the HTML5 document does not.
  • Content of <style> and <script> (raw text elements): with HTML4 on CRuby it becomes CDATA nodes that Loofah's sanitizers escape; see Internals.

The Six Classes

Class Superclass Adds
Loofah::HTML5::Document Nokogiri::HTML5::Document scrub!, text, to_text, comment cleanup at parse time
Loofah::HTML5::DocumentFragment Nokogiri::HTML5::DocumentFragment scrub!, text, to_text, fragment-aware to_s
Loofah::HTML4::Document Nokogiri::HTML4::Document Same as the HTML5 document
Loofah::HTML4::DocumentFragment Nokogiri::HTML4::DocumentFragment Same as the HTML5 fragment
Loofah::XML::Document Nokogiri::XML::Document scrub!
Loofah::XML::DocumentFragment Nokogiri::XML::DocumentFragment scrub!

Because these are real subclasses, is_a? checks against Nokogiri pass:

Loofah.html5_document(html).is_a?(Nokogiri::HTML5::Document)          # => true
Loofah.html5_fragment(html).is_a?(Nokogiri::HTML5::DocumentFragment)  # => true

The XML classes do not get text / to_text overrides or a custom to_s; they behave like plain Nokogiri XML objects apart from scrub!.

Parsing Methods

Method Returns Arguments
Loofah.html5_document(*args) Loofah::HTML5::Document Passed to Nokogiri::HTML5::Document.parse
Loofah.html5_fragment(tags, encoding = nil) Loofah::HTML5::DocumentFragment Markup and optional encoding name
Loofah.html4_document(*args) Loofah::HTML4::Document Passed to Nokogiri::HTML4::Document.parse
Loofah.html4_fragment(tags, encoding = nil) Loofah::HTML4::DocumentFragment Markup and optional encoding name
Loofah.xml_document(*args) Loofah::XML::Document Passed to Nokogiri::XML::Document.parse
Loofah.xml_fragment(tags) Loofah::XML::DocumentFragment Markup only

Each also has a scrub_* shorthand that parses and scrubs in one call, such as Loofah.scrub_html5_fragment(string_or_io, method). The full list is in the API Reference.

Note: The method comments say the fragment shortcuts accept "the same parameters" as Nokogiri's DocumentFragment.parse, but Loofah replaces the fragment classes' parse with its own parse(tags, encoding = nil) (HTML) and parse(tags) (XML). Nokogiri's extra fragment arguments, such as parse options, a context element, or a block, are not supported and passing them raises ArgumentError. The document methods do forward all arguments to Nokogiri.

Input Types and Encodings

All parsing and scrub_* methods accept a String or an IO-like object (anything that responds to read and close, such as a File or StringIO).

For HTML fragments, the encoding is chosen as follows:

  1. The encoding argument, if you pass one.
  2. Otherwise the input's own encoding name, if the input responds to encoding (Ruby strings do).
  3. Otherwise "UTF-8".

The chosen name is assigned to the fragment's internal document:

frag = Loofah.html5_fragment("<div>bar</div>", "US-ASCII")
frag.document.encoding # => "US-ASCII"

XML fragments use the input's encoding when it has one. In practice this means a UTF-8 string goes in and a UTF-8 string comes out of to_s; the test suite checks this for Japanese text with HTML4, HTML5, and XML fragments.

Scrubbing Nodes and Node Sets

Every node and NodeSet inside a Loofah document or fragment also gets scrub!, which lets you sanitize only part of a tree:

doc = Loofah.xml_document(xml)
doc.xpath("//employee").scrub!(employee_scrubber)    # every <employee> subtree
doc.at_xpath("//employee").scrub!(employee_scrubber) # only the first one

An HTML example from the test suite:

doc = Loofah.html5_document(<<~HTML)
  <html><body>
    <div class='scrub'><script>I should be removed</script></div>
    <div class='noscrub'><script>I should remain</script></div>
  </body></html>
HTML

doc.at_css("div.scrub").scrub!(:prune)
doc.to_s =~ /I should remain/     # matches
doc.to_s =~ /I should be removed/ # nil

Note: scrub! on nodes only exists for nodes that belong to a Loofah document or fragment. Nodes from a plain Nokogiri::HTML5.parse result do not have it, because Loofah installs the method through Nokogiri's per-document decorators when a Loofah document is created.

scrub! behaves slightly differently depending on the receiver:

Receiver What is traversed
Document The root element and its subtree (nothing if the document has no root)
Fragment Each top-level child and its subtree
Node That node and its subtree
NodeSet Each node in the set and its subtree

In every case scrub! returns the receiver, so you can chain calls.

What Loofah Changes at Parse Time

Parsing an HTML document with Loofah differs from plain Nokogiri in one way: comments that are direct children of the document (outside the <html> element, for example between the DOCTYPE and <html>) are removed immediately after parsing. These comments are legal in HTML, but they cannot be scrubbed like ordinary nodes, so Loofah drops them. Comments inside the document are untouched until a scrubber processes them.

doc = Loofah.html5_document("<!-- before --><html><body>hi</body></html>")
doc.children.any?(&:comment?) # => false

Fragments and XML documents are parsed exactly as Nokogiri would parse them.

See Also