1:mod:`sgmllib` --- Simple SGML parser
2=====================================
3
4.. module:: sgmllib
5   :synopsis: Only as much of an SGML parser as needed to parse HTML.
6   :deprecated:
7
8.. deprecated:: 2.6
9    The :mod:`sgmllib` module has been removed in Python 3.
10
11.. index:: single: SGML
12
13This module defines a class :class:`SGMLParser` which serves as the basis for
14parsing text files formatted in SGML (Standard Generalized Mark-up Language).
15In fact, it does not provide a full SGML parser --- it only parses SGML insofar
16as it is used by HTML, and the module only exists as a base for the
17:mod:`htmllib` module.  Another HTML parser which supports XHTML and offers a
18somewhat different interface is available in the :mod:`HTMLParser` module.
19
20
21.. class:: SGMLParser()
22
23   The :class:`SGMLParser` class is instantiated without arguments. The parser is
24   hardcoded to recognize the following constructs:
25
26   * Opening and closing tags of the form ``<tag attr="value" ...>`` and
27     ``</tag>``, respectively.
28
29   * Numeric character references of the form ``&#name;``.
30
31   * Entity references of the form ``&name;``.
32
33   * SGML comments of the form ``<!--text-->``.  Note that spaces, tabs, and
34     newlines are allowed between the trailing ``>`` and the immediately preceding
35     ``--``.
36
37A single exception is defined as well:
38
39
40.. exception:: SGMLParseError
41
42   Exception raised by the :class:`SGMLParser` class when it encounters an error
43   while parsing.
44
45   .. versionadded:: 2.1
46
47:class:`SGMLParser` instances have the following methods:
48
49
50.. method:: SGMLParser.reset()
51
52   Reset the instance.  Loses all unprocessed data.  This is called implicitly at
53   instantiation time.
54
55
56.. method:: SGMLParser.setnomoretags()
57
58   Stop processing tags.  Treat all following input as literal input (CDATA).
59   (This is only provided so the HTML tag ``<PLAINTEXT>`` can be implemented.)
60
61
62.. method:: SGMLParser.setliteral()
63
64   Enter literal mode (CDATA mode).
65
66
67.. method:: SGMLParser.feed(data)
68
69   Feed some text to the parser.  It is processed insofar as it consists of
70   complete elements; incomplete data is buffered until more data is fed or
71   :meth:`close` is called.
72
73
74.. method:: SGMLParser.close()
75
76   Force processing of all buffered data as if it were followed by an end-of-file
77   mark.  This method may be redefined by a derived class to define additional
78   processing at the end of the input, but the redefined version should always call
79   :meth:`close`.
80
81
82.. method:: SGMLParser.get_starttag_text()
83
84   Return the text of the most recently opened start tag.  This should not normally
85   be needed for structured processing, but may be useful in dealing with HTML "as
86   deployed" or for re-generating input with minimal changes (whitespace between
87   attributes can be preserved, etc.).
88
89
90.. method:: SGMLParser.handle_starttag(tag, method, attributes)
91
92   This method is called to handle start tags for which either a :meth:`start_tag`
93   or :meth:`do_tag` method has been defined.  The *tag* argument is the name of
94   the tag converted to lower case, and the *method* argument is the bound method
95   which should be used to support semantic interpretation of the start tag. The
96   *attributes* argument is a list of ``(name, value)`` pairs containing the
97   attributes found inside the tag's ``<>`` brackets.
98
99   The *name* has been translated to lower case. Double quotes and backslashes in
100   the *value* have been interpreted, as well as known character references and
101   known entity references terminated by a semicolon (normally, entity references
102   can be terminated by any non-alphanumerical character, but this would break the
103   very common case of ``<A HREF="url?spam=1&eggs=2">`` when ``eggs`` is a valid
104   entity name).
105
106   For instance, for the tag ``<A HREF="http://www.cwi.nl/">``, this method would
107   be called as ``unknown_starttag('a', [('href', 'http://www.cwi.nl/')])``.  The
108   base implementation simply calls *method* with *attributes* as the only
109   argument.
110
111   .. versionadded:: 2.5
112      Handling of entity and character references within attribute values.
113
114
115.. method:: SGMLParser.handle_endtag(tag, method)
116
117   This method is called to handle endtags for which an :meth:`end_tag` method has
118   been defined.  The *tag* argument is the name of the tag converted to lower
119   case, and the *method* argument is the bound method which should be used to
120   support semantic interpretation of the end tag.  If no :meth:`end_tag` method is
121   defined for the closing element, this handler is not called.  The base
122   implementation simply calls *method*.
123
124
125.. method:: SGMLParser.handle_data(data)
126
127   This method is called to process arbitrary data.  It is intended to be
128   overridden by a derived class; the base class implementation does nothing.
129
130
131.. method:: SGMLParser.handle_charref(ref)
132
133   This method is called to process a character reference of the form ``&#ref;``.
134   The base implementation uses :meth:`convert_charref` to convert the reference to
135   a string.  If that method returns a string, it is passed to :meth:`handle_data`,
136   otherwise ``unknown_charref(ref)`` is called to handle the error.
137
138   .. versionchanged:: 2.5
139      Use :meth:`convert_charref` instead of hard-coding the conversion.
140
141
142.. method:: SGMLParser.convert_charref(ref)
143
144   Convert a character reference to a string, or ``None``.  *ref* is the reference
145   passed in as a string.  In the base implementation, *ref* must be a decimal
146   number in the range 0--255.  It converts the code point found using the
147   :meth:`convert_codepoint` method. If *ref* is invalid or out of range, this
148   method returns ``None``.  This method is called by the default
149   :meth:`handle_charref` implementation and by the attribute value parser.
150
151   .. versionadded:: 2.5
152
153
154.. method:: SGMLParser.convert_codepoint(codepoint)
155
156   Convert a code point to a :class:`str` value.  Encodings can be handled here if
157   appropriate, though the rest of :mod:`sgmllib` is oblivious on this matter.
158
159   .. versionadded:: 2.5
160
161
162.. method:: SGMLParser.handle_entityref(ref)
163
164   This method is called to process a general entity reference of the form
165   ``&ref;`` where *ref* is a general entity reference.  It converts *ref* by
166   passing it to :meth:`convert_entityref`.  If a translation is returned, it calls
167   the method :meth:`handle_data` with the translation; otherwise, it calls the
168   method ``unknown_entityref(ref)``. The default :attr:`entitydefs` defines
169   translations for ``&amp;``, ``&apos``, ``&gt;``, ``&lt;``, and ``&quot;``.
170
171   .. versionchanged:: 2.5
172      Use :meth:`convert_entityref` instead of hard-coding the conversion.
173
174
175.. method:: SGMLParser.convert_entityref(ref)
176
177   Convert a named entity reference to a :class:`str` value, or ``None``.  The
178   resulting value will not be parsed.  *ref* will be only the name of the entity.
179   The default implementation looks for *ref* in the instance (or class) variable
180   :attr:`entitydefs` which should be a mapping from entity names to corresponding
181   translations.  If no translation is available for *ref*, this method returns
182   ``None``.  This method is called by the default :meth:`handle_entityref`
183   implementation and by the attribute value parser.
184
185   .. versionadded:: 2.5
186
187
188.. method:: SGMLParser.handle_comment(comment)
189
190   This method is called when a comment is encountered.  The *comment* argument is
191   a string containing the text between the ``<!--`` and ``-->`` delimiters, but
192   not the delimiters themselves.  For example, the comment ``<!--text-->`` will
193   cause this method to be called with the argument ``'text'``.  The default method
194   does nothing.
195
196
197.. method:: SGMLParser.handle_decl(data)
198
199   Method called when an SGML declaration is read by the parser.  In practice, the
200   ``DOCTYPE`` declaration is the only thing observed in HTML, but the parser does
201   not discriminate among different (or broken) declarations.  Internal subsets in
202   a ``DOCTYPE`` declaration are not supported.  The *data* parameter will be the
203   entire contents of the declaration inside the ``<!``...\ ``>`` markup.  The
204   default implementation does nothing.
205
206
207.. method:: SGMLParser.report_unbalanced(tag)
208
209   This method is called when an end tag is found which does not correspond to any
210   open element.
211
212
213.. method:: SGMLParser.unknown_starttag(tag, attributes)
214
215   This method is called to process an unknown start tag.  It is intended to be
216   overridden by a derived class; the base class implementation does nothing.
217
218
219.. method:: SGMLParser.unknown_endtag(tag)
220
221   This method is called to process an unknown end tag.  It is intended to be
222   overridden by a derived class; the base class implementation does nothing.
223
224
225.. method:: SGMLParser.unknown_charref(ref)
226
227   This method is called to process unresolvable numeric character references.
228   Refer to :meth:`handle_charref` to determine what is handled by default.  It is
229   intended to be overridden by a derived class; the base class implementation does
230   nothing.
231
232
233.. method:: SGMLParser.unknown_entityref(ref)
234
235   This method is called to process an unknown entity reference.  It is intended to
236   be overridden by a derived class; the base class implementation does nothing.
237
238Apart from overriding or extending the methods listed above, derived classes may
239also define methods of the following form to define processing of specific tags.
240Tag names in the input stream are case independent; the *tag* occurring in
241method names must be in lower case:
242
243
244.. method:: SGMLParser.start_tag(attributes)
245   :noindex:
246
247   This method is called to process an opening tag *tag*.  It has preference over
248   :meth:`do_tag`.  The *attributes* argument has the same meaning as described for
249   :meth:`handle_starttag` above.
250
251
252.. method:: SGMLParser.do_tag(attributes)
253   :noindex:
254
255   This method is called to process an opening tag *tag*  for which no
256   :meth:`start_tag` method is defined.   The *attributes* argument has the same
257   meaning as described for :meth:`handle_starttag` above.
258
259
260.. method:: SGMLParser.end_tag()
261   :noindex:
262
263   This method is called to process a closing tag *tag*.
264
265Note that the parser maintains a stack of open elements for which no end tag has
266been found yet.  Only tags processed by :meth:`start_tag` are pushed on this
267stack.  Definition of an :meth:`end_tag` method is optional for these tags.  For
268tags processed by :meth:`do_tag` or by :meth:`unknown_tag`, no :meth:`end_tag`
269method must be defined; if defined, it will not be used.  If both
270:meth:`start_tag` and :meth:`do_tag` methods exist for a tag, the
271:meth:`start_tag` method takes precedence.
272
273