Home

News

Introduction

HtmlCleaner is open-source HTML parser written in Java. HTML found on Web is usually dirty, ill-formed and unsuitable for further processing. For any serious consumption of such documents, it is necessary to first clean up the mess and bring the order to tags, attributes and ordinary text. For the given HTML document, HtmlCleaner reorders individual elements and produces well-formed XML. By default, it follows similar rules that the most of web-browsers use in order to create document object model. However, user may provide custom tag and rule set for tag filtering and balancing.

At the time of developing this tool, some open source Java solutions have existed for a long time. However, by the author's experience, they are either not maintained any more or fail to produce well-formed XML in all cases. A few of them make sometimes XML results with unexpected or unstable structure. This was the main motive for starting this project - to create small (JAR file bellow 30K), fast and reliable tool that will always produce well-formed XML.

Setting cleaner's behavior

HtmlCleaner can be tuned using several parameters. They are briefly described in the following table:

Command line use

HtmlCleaner can be called from the command line with the following syntax:

    java -jar htmlcleanerXX.jar src = <url | file> [incharset = <charset>] 
                                [dest = <file>] [outcharset = <charset>] 
                                [options...]
    

where options include:

    outputtype = simple | compact | browser-compact | pretty
    advancedxmlescape = true | false
    usecdata = true | false
    specialentities = true | false
    unicodechars = true | false
    omitunknowntags = true | false
    treatunknowntagsascontent = true | false
    omitdeprtags = true | false
    treatdeprtagsascontent = true | false
    omitcomments = true | false
    omitxmldecl = true | false
    omitdoctypedecl = true | false
    omitxmlnsatt = true | false
    omithtmlenvelope = true | false
    useemptyelementtags = true | false
    allowmultiwordattributes = true | false
    allowhtmlinsideattributes = true | false
    ignoreqe = true | false
    namespacesaware = true | false
    hyphenreplacement = <string value>
    prunetags = <string value>
    

Note: in order to make difference between URLs and files, URS's must begin with http://

Ant use

Ensure that HtmlCleaner JAR file is at the Ant's class path. Create Ant task in the following way:

    <taskdef name="mytask" classname="org.htmlcleaner.HtmlCleanerForAnt">
    ....
    <target name="...">
        <mytask [src = "..."] 
                [incharset = "..."]
                [dest = "..."] 
                [outcharset = "..."] 
                [outputtype = "simple" | "compact" | "pretty"]
                [advancedxmlescape = "true" | "false"]
                [usecdata = "true" | "false"]
                [specialentities = "true" | "false"]
                [unicodechars = "true" | "false"]
                [omitunknowntags = "true" | "false"]
                [treatunknowntagsascontent = "true" | "false"]
                [omitdeprtags = "true" | "false"]
                [treatdeprtagsascontent = "true" | "false"]
                [omitcomments = "true" | "false"]
                [omitxmldecl = "true" | "false"]
                [omitdoctypedecl = "true" | "false"]
                [omitxmlnsatt = "true" | "false"]
                [omithtmlenvelope = "true" | "false"]
                [useemptyelementtags = "true" | "false"]
                [allowmultiwordattributes = "true" | "false"]
                [allowhtmlinsideattributes = "true" | "false"]
                [ignoreqe = "true" | "false"]
                [namespacesaware = "true" | "false"]
                [hyphenreplacement = "..."]>
                [prunetags = "..."]>
 
            .... optional HTML code ....
            
        </mytask>
    </target>
    

Note: in order to make difference between URLs and files, URS's must begin with http:// in src attribute. If src attribute is not specified, HTML from the task's body is used.

Java code use

Typical usage from Java code is the following:

    // one of several constructors
    HtmlCleaner cleaner = new HtmlCleaner(...);
    
    // optionally, set cleaner's behavior     
    cleaner.setXXX(...)
    
    // calls cleaning process
    cleaner.clean();
    
    // writes resulting XML to string, file or any output stream...
    cleaner.writeXmlXXX(...);

    // ...or create DOM object
    // org.w3c.dom.Document myDoc = cleaner.createDOM();

    // ...or create JDom object
    org.jdom.Document myJDoc = cleaner.createJDom();

    // ... or write to your own serializer
    writeXml(myXmlSerializer);

    // ... or just take resulting node and do whatever you want with it
    org.htmlcleaner.TagNode rootNode = claner.getRootNode();
    

Implementing custom tag info set

HtmlCleaner implements default HTML tag set and rules for their balancing, that mimics the browsers' behavior. For example, someone may not like the rule that implicit TBODY is inserted before TR in the HTML table. In order to make custom rules and set of tags, do the following:

• Implement interface org.htmlcleaner.ITagInfoProvider
• Use instance of implemented class in some of HtmlCelaner constructors.

See Java API for more details, and download source code to check how default HTML provider is implemented.