1 TagSoup - Just Keep On Truckin' 2 3 Introduction 4 5 This is the home page of TagSoup, a SAX-compliant parser written in 6 Java that, instead of parsing well-formed or valid XML, parses HTML as 7 it is found in the wild: [1]poor, nasty and brutish, though quite often 8 far from short. TagSoup is designed for people who have to process this 9 stuff using some semblance of a rational application design. By 10 providing a SAX interface, it allows standard XML tools to be applied 11 to even the worst HTML. TagSoup also includes a command-line processor 12 that reads HTML files and can generate either clean HTML or well-formed 13 XML that is a close approximation to XHTML. 14 15 This is also the README file packaged with TagSoup. 16 17 TagSoup is free and Open Source software. As of version 1.2, it is 18 licensed under the [2]Apache License, Version 2.0, which allows 19 proprietary re-use as well as use with GPL 3.0 or GPL 2.0-or-later 20 projects. (If anyone needs a GPL 2.0 license for a GPL 2.0-only 21 project, feel free to ask.) 22 23 Warning: TagSoup will not build on stock Java 5.x or 6.x! 24 25 Due to a bug in the versions of Xalan shipped with Java 5.x and 6.x, 26 TagSoup will not build out of the box. You need to retrieve [3]Saxon 27 6.5.5, which does not have the bug. Unpack the zipfile in an empty 28 directory and copy the saxon.jar and saxon-xml-apis.jar files to 29 $ANT_HOME/lib. The Ant build process for TagSoup will then notice that 30 Saxon is available and use it instead. 31 32 TagSoup 1.2 released 33 34 There are a great many changes, most of them fixes for long-standing 35 bugs, in this release. Only the most important are listed here; for the 36 rest, see the CHANGES file in the source distribution. Very special 37 thanks to Jojo Dijamco, whose intensive efforts at debugging made this 38 release a usable upgrade rather than a useless mass of undetected bugs. 39 * As noted above, I have changed the license to Apache 2.0. 40 * The default content model for bogons (unknown elements) is now ANY 41 rather than EMPTY. This is a breaking change, which I have done 42 only because there was so much demand for it. It can be undone on 43 the command line with the --emptybogons switch, or programmatically 44 with parser.setFeature(Parser.emptyBogonsFeature, true). 45 * The processing of entity references in attribute values has finally 46 been fixed to do what browsers do. That is, a reference is only 47 recognized if it is properly terminated by a semicolon; otherwise 48 it is treated as plain text. This means that URIs like 49 foo?cdown=32&cup=42 are no longer seen as containing an instance of 50 the )U character (whose name happens to be cup). 51 * Several new switches have been added: 52 + --doctype-system and --doctype-public force a DOCTYPE 53 declaration to be output and allow setting the system and 54 public identifiers. 55 + --standalone and --version allow control of the XML 56 declaration that is output. (Note that TagSoup's XML output is 57 always version 1.0, even if you use --version=1.1.) 58 + --norootbogons causes unknown elements not to be allowed as 59 the document root element. Instead, they are made children of 60 the default root element (the html element for HTML). 61 * The TagSoup core now supports character entities with values above 62 U+FFFF. As a consequence, the HTML schema now supports all 2,210 63 standard character entities from the [4]2007-12-14 draft of XML 64 Entity Definitions for Characters, except the 94 which require more 65 than one Unicode character to represent. 66 * The SAX events startPrefixMapping and endPrefixMapping are now 67 being reported for all cases of foreign elements and attributes. 68 * All bugs around newline processing on Windows should now be gone. 69 * A number of content models have been loosened to allow elements to 70 appear in new and non-standard (but commonly found) places. In 71 particular, tables are now allowed inside paragraphs, against the 72 letter of the W3C specification. 73 * Since the span element is intended for fine control of appearance 74 using CSS, it should never have been a restartable element. This 75 very long-standing bug has now been fixed. 76 * The following non-standard elements are now at least partly 77 supported: bgsound, blink, canvas, comment, listing, marquee, nobr, 78 rbc, rb, rp, rtc, rt, ruby, wbr, xmp. 79 * In HTML output mode, boolean attributes like checked are now output 80 as such, rather than in XML style as checked="checked". 81 * Runs of < characters such as << and <<< are now handled correctly 82 in text rather than being transformed into extremely bogus 83 start-tags. 84 85 [5]Download the TagSoup 1.2 jar file here. It's about 87K long. 86 [6]Download the full TagSoup 1.2 source here. If you don't have zip, 87 you can use jar to unpack it. 88 [7]Download the current CHANGES file here. 89 90 TagSoup 1.1 released 91 92 TagSoup 1.1 adds Tatu Saloranta's JAXP support for TagSoup. To use 93 TagSoup within the JAXP framework (which is not something I necessarily 94 recommend, but it is part of the Java XML platform), you can create a 95 SAXParser by calling 96 org.ccil.cowan.tagsoup.jaxp.SAXParserImpl.newInstance(). You can also 97 set the system property javax.xml.parsers.SAXParserFactory to 98 org.ccil.cowan.tagsoup.jaxp.SAXFactoryImpl, but be aware that doing 99 this will cause all JAXP-based XML parsing to go through TagSoup, which 100 is a Bad Thing if your application also reads XML documents. 101 102 What TagSoup does 103 104 TagSoup is designed as a parser, not a whole application; it isn't 105 intended to permanently clean up bad HTML, as [8]HTML Tidy does, only 106 to parse it on the fly. Therefore, it does not convert presentation 107 HTML to CSS or anything similar. It does guarantee well-structured 108 results: tags will wind up properly nested, default attributes will 109 appear appropriately, and so on. 110 111 The semantics of TagSoup are as far as practical those of actual HTML 112 browsers. In particular, never, never will it throw any sort of syntax 113 error: the TagSoup motto is [9]"Just Keep On Truckin'". But there's 114 much, much more. For example, if the first tag is LI, it will supply 115 the application with enclosing HTML, BODY, and UL tags. Why UL? Because 116 that's what browsers assume in this situation. For the same reason, 117 overlapping tags are correctly restarted whenever possible: text like: 118This is <B>bold, <I>bold italic, </b>italic, </i>normal text 119 120 gets correctly rewritten as: 121This is <b>bold, <i>bold italic, </i></b><i>italic, </i>normal text. 122 123 By intention, TagSoup is small and fast. It does not depend on the 124 existence of any framework other than SAX, and should be able to work 125 with any framework that can accept SAX parsers. In particular, [10]XOM 126 is known to work. 127 128 You can replace the low-level HTML scanner with one based on Sean 129 McGrath's [11]PYX format (very close to James Clark's ESIS format). You 130 can also supply an AutoDetector that peeks at the incoming byte stream 131 and guesses a character encoding for it. Otherwise, the platform 132 default is used. If you need an autodetector of character sets, 133 consider trying to adapt the [12]Mozilla one; if you succeed, let me 134 know. 135 136 Note: TagSoup in Java 1.1 137 138 If you go through the TagSoup source and replace all references to 139 HashMap with Hashtable and recompile, TagSoup will work fine in Java 140 1.1 VMs. Thanks to Thorbj�rn Vinne for this discovery. 141 142 The TSaxon XSLT-for-HTML processor 143 144 [13]I am also distributing [14]TSaxon, a repackaging of version 6.5.5 145 of Michael Kay's Saxon XSLT version 1.0 implementation that includes 146 TagSoup. TSaxon is a drop-in replacement for Saxon, and can be used to 147 process either HTML or XML documents with XSLT stylesheets. 148 149 TagSoup as a stand-alone program 150 151 It is possible to run TagSoup as a program by saying java -jar 152 tagsoup-1.0.1 [option ...] [file ...]. Files mentioned on the command 153 line will be parsed individually. If no files are specified, the 154 standard input is read. 155 156 The following options are understood: 157 158 --files 159 Output into individual files, with html extensions changed to 160 xhtml. Otherwise, all output is sent to the standard output. 161 162 --html 163 Output is in clean HTML: the XML declaration is suppressed, as 164 are end-tags for the known empty elements. 165 166 --omit-xml-declaration 167 The XML declaration is suppressed. 168 169 --method=html 170 End-tags for the known empty HTML elements are suppressed. 171 172 --doctype-system=systemid 173 Forces the output of a DOCTYPE declaration with the specified 174 systemid. 175 176 --doctype-public=publicid 177 Forces the output of a DOCTYPE declaration with the specified 178 publicid. 179 180 --version=version 181 Sets the version string in the XML declaration. 182 183 --standalone=[yes|no] 184 Sets the standalone declaration to yes or no. 185 186 --pyx 187 Output is in PYX format. 188 189 --pyxin 190 Input is in PYXoid format (need not be well-formed). 191 192 --nons 193 Namespaces are suppressed. Normally, all elements are in the 194 XHTML 1.x namespace, and all attributes are in no namespace. 195 196 --nobogons 197 Bogons (unknown elements) are suppressed. 198 199 --nodefaults 200 suppress default attribute values 201 202 --nocolons 203 change explicit colons in element and attribute names to 204 underscores 205 206 --norestart 207 don't restart any normally restartable elements 208 209 --ignorable 210 output whitespace in elements with element-only content 211 212 --emptybogons 213 Bogons are given a content model of EMPTY rather than ANY. 214 215 --any 216 Bogons are given a content model of ANY rather than EMPTY 217 (default). 218 219 --norootbogons 220 Don't allow bogons to be root elements; make them subordinate to 221 the root. 222 223 --lexical 224 Pass through HTML comments and DOCTYPE declarations. Has no 225 effect when output is in PYX format. 226 227 --reuse 228 Reuse a single instance of TagSoup parser throughout. Normally, 229 a new one is instantiated for each input file. 230 231 --nocdata 232 Change the content models of the script and style elements to 233 treat them as ordinary #PCDATA (text-only) elements, as in 234 XHTML, rather than with the special CDATA content model. 235 236 --encoding=encoding 237 Specify the input encoding. The default is the Java platform 238 default. 239 240 --output-encoding=encoding 241 Specify the output encoding. The default is the Java platform 242 default. 243 244 --help 245 Print help. 246 247 --version 248 Print the version number. 249 250 SAX features and properties 251 252 TagSoup supports the following SAX features in addition to the standard 253 ones: 254 255 http://www.ccil.org/~cowan/tagsoup/features/ignore-bogons 256 A value of "true" indicates that the parser will ignore unknown 257 elements. 258 259 http://www.ccil.org/~cowan/tagsoup/features/bogons-empty 260 A value of "true" indicates that the parser will give unknown 261 elements a content model of EMPTY; a value of "false", a content 262 model of ANY. 263 264 http://www.ccil.org/~cowan/tagsoup/features/root-bogons 265 A value of "true" indicates that the parser will allow unknown 266 elements to be the root of the output document. 267 268 http://www.ccil.org/~cowan/tagsoup/features/default-attributes 269 A value of "true" indicates that the parser will return default 270 attribute values for missing attributes that have default 271 values. 272 273 http://www.ccil.org/~cowan/tagsoup/features/translate-colons 274 A value of "true" indicates that the parser will translate 275 colons into underscores in names. 276 277 http://www.ccil.org/~cowan/tagsoup/features/restart-elements 278 A value of "true" indicates that the parser will attempt to 279 restart the restartable elements. 280 281 http://www.ccil.org/~cowan/tagsoup/features/ignorable-whitespace 282 A value of "true" indicates that the parser will transmit 283 whitespace in element-only content via the SAX 284 ignorableWhitespace callback. Normally this is not done, because 285 HTML is an SGML application and SGML suppresses such whitespace. 286 287 http://www.ccil.org/~cowan/tagsoup/features/cdata-elements 288 A value of "true" indicates that the parser will process the 289 script and style elements (or any elements with type='cdata' in 290 the TSSL schema) as SGML CDATA elements (that is, no markup is 291 recognized except the matching end-tag). 292 293 TagSoup supports the following SAX properties in addition to the 294 standard ones: 295 296 http://www.ccil.org/~cowan/tagsoup/properties/scanner 297 Specifies the Scanner object this parser uses. 298 299 http://www.ccil.org/~cowan/tagsoup/properties/schema 300 Specifies the Schema object this parser uses. 301 302 http://www.ccil.org/~cowan/tagsoup/properties/auto-detector 303 Specifies the AutoDetector (for encoding detection) this parser 304 uses. 305 306 More information 307 308 I gave a presentation (a nocturne, so it's not on the schedule) at 309 [15]Extreme Markup Languages 2004 about TagSoup, updated from the one 310 presented in 2002 at the New York City XML SIG and at XML 2002. This is 311 the main high-level documentation about how TagSoup works. Formats: 312 [16]OpenDocument [17]Powerpoint [18]PDF. 313 314 I also had people add [19]"evil" HTML to a large poster so that I could 315 [20]clean it up; View Source is probably more useful than ordinary 316 browsing. The original instructions were: 317 318 SOUPE DE BALISES (BE EVIL)! 319 Ecritez une balise ouvrante (sans attributs) 320 ou fermante HTML ici, s.v.p. 321 322 There is a [21]tagsoup-friends mailing list hosted at [22]Yahoo Groups. 323 You can [23]join via the Web, or by sending a blank email to 324 [24]tagsoup-friends-subscribe@yahoogroups.com. The [25]archives are 325 open to all. 326 327 Online TagSoup processing for publicly accessible HTML documents is now 328 [26]available courtesy of Leigh Dodds. 329 330References 331 332 1. http://oregonstate.edu/instruct/phl302/texts/hobbes/leviathan-c.html 333 2. http://opensource.org/licenses/apache2.0.php 334 3. http://prdownloads.sourceforge.net/saxon/saxon6-5-5.zip 335 4. http://www.w3.org/TR/2007/WD-xml-entity-names-20071214 336 5. http://home.ccil.org/~cowan/XML/tagsoup/tagsoup-1.2.jar 337 6. http://home.ccil.org/~cowan/XML/tagsoup/tagsoup-1.2-src.zip 338 7. http://home.ccil.org/~cowan/XML/tagsoup/CHANGES 339 8. http://tidy.sf.net/ 340 9. http://www.crumbmuseum.com/truckin.html 341 10. http://www.cafeconleche.org/XOM 342 11. http://gnosis.cx/publish/programming/xml_matters_17.html 343 12. http://jchardet.sourceforge.net/ 344 13. http://www.ccil.org/~cowan 345 14. http://home.ccil.org/~cowan/XML/tagsoup/tsaxon 346 15. http://www.extrememarkup.com/extreme/2004 347 16. http://home.ccil.org/~cowan/XML/tagsoup/tagsoup.odp 348 17. http://home.ccil.org/~cowan/XML/tagsoup/tagsoup.ppt 349 18. http://home.ccil.org/~cowan/XML/tagsoup/tagsoup.pdf 350 19. http://home.ccil.org/~cowan/XML/tagsoup/extreme.html 351 20. http://home.ccil.org/~cowan/XML/tagsoup/extreme.xhtml 352 21. http://groups.yahoo.com/group/tagsoup-friends 353 22. http://groups.yahoo.com/ 354 23. http://groups.yahoo.com/group/tagsoup-friends/join 355 24. mailto:tagsoup-friends-subscribe@yahoogroups.com 356 25. http://groups.yahoo.com/group/tagsoup-friends/messages 357 26. http://xmlarmyknife.org/docs/xhtml/tagsoup/ 358