cpython-lru_cache-weakref: Lib/html/parser.py annotate

annotate Lib/html/parser.py @ 87690:66c1c9f32567 default tip

Add use_weakref option to functools.lru_cache The use_weakref option makes lru_cache create weak references to the function arguments when creating the cache key, which prevents them from staying in the memory forever even after all other references to them are gone. This is a proof of concept for http://bugs.python.org/issue19859

author	Radomir Dopieralski <devel@sheep.art.pl>
date	Tue, 03 Dec 2013 16:02:48 +0100
parents	1575f2dd08c4
children

rev	line source
18984 00935031b373 Change some comments into docstrings. Fred Drake <fdrake@acm.org> parents: 17934 diff changeset	1 """A parser for HTML and XHTML."""
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	2
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	3 # This file is based on sgmllib.py, but the API is slightly different.
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	4
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	5 # XXX There should be a way to distinguish between PCDATA (parsed
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	6 # character data -- the normal case), RCDATA (replaceable character
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	7 # data -- only char and entity references and end tags are special)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	8 # and CDATA (character data -- only end tags are special).
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	9
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	10
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	11 import re
77630 8dd2f5754b2f #15114: the strict mode of HTMLParser and the HTMLParseError exception are deprecated now that the parser is able to parse invalid markup. Ezio Melotti <ezio.melotti@gmail.com> parents: 76414 diff changeset	12 import warnings
87275 7b9235852b3b #2927: Added the unescape() function to the html module. Ezio Melotti <ezio.melotti@gmail.com> parents: 86990 diff changeset	13 import _markupbase
7b9235852b3b #2927: Added the unescape() function to the html module. Ezio Melotti <ezio.melotti@gmail.com> parents: 86990 diff changeset	14
7b9235852b3b #2927: Added the unescape() function to the html module. Ezio Melotti <ezio.melotti@gmail.com> parents: 86990 diff changeset	15 from html import unescape
7b9235852b3b #2927: Added the unescape() function to the html module. Ezio Melotti <ezio.melotti@gmail.com> parents: 86990 diff changeset	16
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	17
83581 1f7ce8af3356 #14679: add an __all__ (that contains only HTMLParser) to html.parser. Ezio Melotti <ezio.melotti@gmail.com> parents: 77742 diff changeset	18 __all__ = ['HTMLParser']
1f7ce8af3356 #14679: add an __all__ (that contains only HTMLParser) to html.parser. Ezio Melotti <ezio.melotti@gmail.com> parents: 77742 diff changeset	19
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	20 # Regular expressions used for parsing
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	21
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	22 interesting_normal = re.compile('[&<]')
19682 8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases Fred Drake <fdrake@acm.org> parents: 19410 diff changeset	23 incomplete = re.compile('&[a-zA-Z#]')
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	24
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	25 entityref = re.compile('&([a-zA-Z][-.a-zA-Z0-9]*)[^a-zA-Z0-9]')
18984 00935031b373 Change some comments into docstrings. Fred Drake <fdrake@acm.org> parents: 17934 diff changeset	26 charref = re.compile('&#(?:[0-9]+\|[xX][0-9a-fA-F]+)[^0-9a-fA-F]')
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	27
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	28 starttagopen = re.compile('<[a-zA-Z]')
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	29 piclose = re.compile('>')
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	30 commentclose = re.compile(r'--\s*>')
86989 9b9d188ed549 #19480: HTMLParser now accepts all valid start-tag names as defined by the HTML5 standard. Ezio Melotti <ezio.melotti@gmail.com> parents: 83586 diff changeset	31 # Note:
9b9d188ed549 #19480: HTMLParser now accepts all valid start-tag names as defined by the HTML5 standard. Ezio Melotti <ezio.melotti@gmail.com> parents: 83586 diff changeset	32 # 1) the strict attrfind isn't really strict, but we can't make it
9b9d188ed549 #19480: HTMLParser now accepts all valid start-tag names as defined by the HTML5 standard. Ezio Melotti <ezio.melotti@gmail.com> parents: 83586 diff changeset	33 # correctly strict without breaking backward compatibility;
9b9d188ed549 #19480: HTMLParser now accepts all valid start-tag names as defined by the HTML5 standard. Ezio Melotti <ezio.melotti@gmail.com> parents: 83586 diff changeset	34 # 2) if you change tagfind/attrfind remember to update locatestarttagend too;
9b9d188ed549 #19480: HTMLParser now accepts all valid start-tag names as defined by the HTML5 standard. Ezio Melotti <ezio.melotti@gmail.com> parents: 83586 diff changeset	35 # 3) if you change tagfind/attrfind and/or locatestarttagend the parser will
9b9d188ed549 #19480: HTMLParser now accepts all valid start-tag names as defined by the HTML5 standard. Ezio Melotti <ezio.melotti@gmail.com> parents: 83586 diff changeset	36 # explode, so don't do it.
76414 ba4baaddac8d #14538: HTMLParser can now parse correctly start tags that contain a bare /. Ezio Melotti <ezio.melotti@gmail.com> parents: 75117 diff changeset	37 tagfind = re.compile('([a-zA-Z][-.a-zA-Z0-9:_])(?:\s\|/(?!>))')
74901 df5e5eea7833 #13993: HTMLParser is now able to handle broken end tags when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 74859 diff changeset	38 # see http://www.w3.org/TR/html5/tokenization.html#tag-open-state
df5e5eea7833 #13993: HTMLParser is now able to handle broken end tags when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 74859 diff changeset	39 # and http://www.w3.org/TR/html5/tokenization.html#tag-name-state
86989 9b9d188ed549 #19480: HTMLParser now accepts all valid start-tag names as defined by the HTML5 standard. Ezio Melotti <ezio.melotti@gmail.com> parents: 83586 diff changeset	40 tagfind_tolerant = re.compile('([a-zA-Z][^\t\n\r\f />\x00])(?:\s\|/(?!>))')
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	41 attrfind = re.compile(
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	42 r'\s([a-zA-Z_][-.:a-zA-Z_0-9])(\s=\s'
69189 225400cb6e84 #7311: fix html.parser to accept non-ASCII attribute values. Ezio Melotti parents: 68257 diff changeset	43 r'(\'[^\']\'\|"[^"]"\|[^\s"\'=<>`]*))?')
66623 1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	44 attrfind_tolerant = re.compile(
76414 ba4baaddac8d #14538: HTMLParser can now parse correctly start tags that contain a bare /. Ezio Melotti <ezio.melotti@gmail.com> parents: 75117 diff changeset	45 r'((?<=[\'"\s/])[^\s/>][^\s/=>])(\s=+\s*'
75117 2d16048b10cd HTMLParser is now able to handle slashes in the start tag. Ezio Melotti <ezio.melotti@gmail.com> parents: 74913 diff changeset	46 r'(\'[^\']\'\|"[^"]"\|(?![\'"])[^>\s]))?(?:\s\|/(?!>))')
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	47 locatestarttagend = re.compile(r"""
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	48 <[a-zA-Z][-.a-zA-Z0-9:_]* # tag name
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	49 (?:\s+ # whitespace before attribute name
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	50 (?:[a-zA-Z_][-.:a-zA-Z0-9_]* # attribute name
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	51 (?:\s=\s # value indicator
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	52 (?:'[^']*' # LITA-enclosed value
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	53 \|\"[^\"]*\" # LIT-enclosed value
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	54 \|[^'\">\s]+ # bare value
35507 15a8975f7fa9 Reverting previous checkin. This breaks too much of HTMLParser to be applied Georg Brandl <georg@python.org> parents: 35499 diff changeset	55 )
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	56 )?
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	57 )
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	58 )*
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	59 \s* # trailing whitespace
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	60 """, re.VERBOSE)
66623 1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	61 locatestarttagend_tolerant = re.compile(r"""
86989 9b9d188ed549 #19480: HTMLParser now accepts all valid start-tag names as defined by the HTML5 standard. Ezio Melotti <ezio.melotti@gmail.com> parents: 83586 diff changeset	62 <[a-zA-Z][^\t\n\r\f />\x00]* # tag name
75117 2d16048b10cd HTMLParser is now able to handle slashes in the start tag. Ezio Melotti <ezio.melotti@gmail.com> parents: 74913 diff changeset	63 (?:[\s/]* # optional whitespace before attribute name
2d16048b10cd HTMLParser is now able to handle slashes in the start tag. Ezio Melotti <ezio.melotti@gmail.com> parents: 74913 diff changeset	64 (?:(?<=['"\s/])[^\s/>][^\s/=>]* # attribute name
73547 16ed15ff0d7c #1745761, #755670, #13357, #12629, #1200313: improve attribute handling in HTMLParser. Ezio Melotti <ezio.melotti@gmail.com> parents: 73274 diff changeset	65 (?:\s=+\s # value indicator
66623 1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	66 (?:'[^']*' # LITA-enclosed value
73547 16ed15ff0d7c #1745761, #755670, #13357, #12629, #1200313: improve attribute handling in HTMLParser. Ezio Melotti <ezio.melotti@gmail.com> parents: 73274 diff changeset	67 \|"[^"]*" # LIT-enclosed value
16ed15ff0d7c #1745761, #755670, #13357, #12629, #1200313: improve attribute handling in HTMLParser. Ezio Melotti <ezio.melotti@gmail.com> parents: 73274 diff changeset	68 \|(?!['"])[^>\s]* # bare value
66623 1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	69 )
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	70 (?:\s,) # possibly followed by a comma
75117 2d16048b10cd HTMLParser is now able to handle slashes in the start tag. Ezio Melotti <ezio.melotti@gmail.com> parents: 74913 diff changeset	71 )?(?:\s\|/(?!>))*
73547 16ed15ff0d7c #1745761, #755670, #13357, #12629, #1200313: improve attribute handling in HTMLParser. Ezio Melotti <ezio.melotti@gmail.com> parents: 73274 diff changeset	72 )*
16ed15ff0d7c #1745761, #755670, #13357, #12629, #1200313: improve attribute handling in HTMLParser. Ezio Melotti <ezio.melotti@gmail.com> parents: 73274 diff changeset	73 )?
66623 1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	74 \s* # trailing whitespace
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	75 """, re.VERBOSE)
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	76 endendtag = re.compile('>')
73274 a6f2244b251f #670664: Fix HTMLParser to correctly handle the content of ``<script>...</script>`` and ``<style>...</style>``. Ezio Melotti <ezio.melotti@gmail.com> parents: 73159 diff changeset	77 # the HTML 5 spec, section 8.1.2.2, doesn't allow spaces between
a6f2244b251f #670664: Fix HTMLParser to correctly handle the content of ``<script>...</script>`` and ``<style>...</style>``. Ezio Melotti <ezio.melotti@gmail.com> parents: 73159 diff changeset	78 # </ and the tag name, so maybe this should be fixed
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	79 endtagfind = re.compile('</\s([a-zA-Z][-.a-zA-Z0-9:_])\s*>')
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	80
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	81
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	82 class HTMLParseError(Exception):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	83 """Exception raised for all parse errors."""
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	84
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	85 def __init__(self, msg, position=(None, None)):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	86 assert msg
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	87 self.msg = msg
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	88 self.lineno = position[0]
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	89 self.offset = position[1]
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	90
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	91 def __str__(self):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	92 result = self.msg
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	93 if self.lineno is not None:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	94 result = result + ", at line %d" % self.lineno
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	95 if self.offset is not None:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	96 result = result + ", column %d" % (self.offset + 1)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	97 return result
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	98
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	99
87437 1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	100 _default_sentinel = object()
86842 0a56709eb798 #15114: The html.parser module now raises a DeprecationWarning when the strict argument of HTMLParser or the HTMLParser.error method are used. Ezio Melotti <ezio.melotti@gmail.com> parents: 83587 diff changeset	101
44212 c6fb393b6c81 rename markupbase to _markupbase Fred Drake <fdrake@acm.org> parents: 41585 diff changeset	102 class HTMLParser(_markupbase.ParserBase):
18984 00935031b373 Change some comments into docstrings. Fred Drake <fdrake@acm.org> parents: 17934 diff changeset	103 """Find tags and other markup and call handler functions.
00935031b373 Change some comments into docstrings. Fred Drake <fdrake@acm.org> parents: 17934 diff changeset	104
00935031b373 Change some comments into docstrings. Fred Drake <fdrake@acm.org> parents: 17934 diff changeset	105 Usage:
00935031b373 Change some comments into docstrings. Fred Drake <fdrake@acm.org> parents: 17934 diff changeset	106 p = HTMLParser()
00935031b373 Change some comments into docstrings. Fred Drake <fdrake@acm.org> parents: 17934 diff changeset	107 p.feed(data)
00935031b373 Change some comments into docstrings. Fred Drake <fdrake@acm.org> parents: 17934 diff changeset	108 ...
00935031b373 Change some comments into docstrings. Fred Drake <fdrake@acm.org> parents: 17934 diff changeset	109 p.close()
17932 ce7f0e380830 Removed incorrect comment left over from sgmllib.py. Guido van Rossum <guido@python.org> parents: 17853 diff changeset	110
18984 00935031b373 Change some comments into docstrings. Fred Drake <fdrake@acm.org> parents: 17934 diff changeset	111 Start tags are handled by calling self.handle_starttag() or
00935031b373 Change some comments into docstrings. Fred Drake <fdrake@acm.org> parents: 17934 diff changeset	112 self.handle_startendtag(); end tags by self.handle_endtag(). The
00935031b373 Change some comments into docstrings. Fred Drake <fdrake@acm.org> parents: 17934 diff changeset	113 data between tags is passed from the parser to the derived class
00935031b373 Change some comments into docstrings. Fred Drake <fdrake@acm.org> parents: 17934 diff changeset	114 by calling self.handle_data() with the data as argument (the data
87437 1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	115 may be split up in arbitrary chunks). If convert_charrefs is
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	116 True the character references are converted automatically to the
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	117 corresponding Unicode character (and self.handle_data() is no
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	118 longer split in chunks), otherwise they are passed by calling
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	119 self.handle_entityref() or self.handle_charref() with the string
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	120 containing respectively the named or numeric reference as the
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	121 argument.
18984 00935031b373 Change some comments into docstrings. Fred Drake <fdrake@acm.org> parents: 17934 diff changeset	122 """
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	123
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	124 CDATA_CONTENT_ELEMENTS = ("script", "style")
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	125
87437 1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	126 def __init__(self, strict=_default_sentinel, *,
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	127 convert_charrefs=_default_sentinel):
66623 1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	128 """Initialize and reset this instance.
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	129
87437 1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	130 If convert_charrefs is True (default: False), all character references
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	131 are automatically converted to the corresponding Unicode characters.
77630 8dd2f5754b2f #15114: the strict mode of HTMLParser and the HTMLParseError exception are deprecated now that the parser is able to parse invalid markup. Ezio Melotti <ezio.melotti@gmail.com> parents: 76414 diff changeset	132 If strict is set to False (the default) the parser will parse invalid
8dd2f5754b2f #15114: the strict mode of HTMLParser and the HTMLParseError exception are deprecated now that the parser is able to parse invalid markup. Ezio Melotti <ezio.melotti@gmail.com> parents: 76414 diff changeset	133 markup, otherwise it will raise an error. Note that the strict mode
86842 0a56709eb798 #15114: The html.parser module now raises a DeprecationWarning when the strict argument of HTMLParser or the HTMLParser.error method are used. Ezio Melotti <ezio.melotti@gmail.com> parents: 83587 diff changeset	134 and argument are deprecated.
66623 1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	135 """
87437 1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	136 if strict is not _default_sentinel:
86842 0a56709eb798 #15114: The html.parser module now raises a DeprecationWarning when the strict argument of HTMLParser or the HTMLParser.error method are used. Ezio Melotti <ezio.melotti@gmail.com> parents: 83587 diff changeset	137 warnings.warn("The strict argument and mode are deprecated.",
77630 8dd2f5754b2f #15114: the strict mode of HTMLParser and the HTMLParseError exception are deprecated now that the parser is able to parse invalid markup. Ezio Melotti <ezio.melotti@gmail.com> parents: 76414 diff changeset	138 DeprecationWarning, stacklevel=2)
86842 0a56709eb798 #15114: The html.parser module now raises a DeprecationWarning when the strict argument of HTMLParser or the HTMLParser.error method are used. Ezio Melotti <ezio.melotti@gmail.com> parents: 83587 diff changeset	139 else:
0a56709eb798 #15114: The html.parser module now raises a DeprecationWarning when the strict argument of HTMLParser or the HTMLParser.error method are used. Ezio Melotti <ezio.melotti@gmail.com> parents: 83587 diff changeset	140 strict = False # default
66623 1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	141 self.strict = strict
87437 1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	142 if convert_charrefs is _default_sentinel:
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	143 convert_charrefs = False # default
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	144 warnings.warn("The value of convert_charrefs will become True in "
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	145 "3.5. You are encouraged to set the value explicitly.",
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	146 DeprecationWarning, stacklevel=2)
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	147 self.convert_charrefs = convert_charrefs
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	148 self.reset()
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	149
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	150 def reset(self):
18984 00935031b373 Change some comments into docstrings. Fred Drake <fdrake@acm.org> parents: 17934 diff changeset	151 """Reset this instance. Loses all unprocessed data."""
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	152 self.rawdata = ''
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	153 self.lasttag = '???'
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	154 self.interesting = interesting_normal
73274 a6f2244b251f #670664: Fix HTMLParser to correctly handle the content of ``<script>...</script>`` and ``<style>...</style>``. Ezio Melotti <ezio.melotti@gmail.com> parents: 73159 diff changeset	155 self.cdata_elem = None
44212 c6fb393b6c81 rename markupbase to _markupbase Fred Drake <fdrake@acm.org> parents: 41585 diff changeset	156 _markupbase.ParserBase.reset(self)
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	157
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	158 def feed(self, data):
70407 829c541033ac Fix display of html.parser.HTMLParser.feed docstring Éric Araujo <merwok@netwok.org> parents: 67189 diff changeset	159 r"""Feed data to the parser.
18984 00935031b373 Change some comments into docstrings. Fred Drake <fdrake@acm.org> parents: 17934 diff changeset	160
00935031b373 Change some comments into docstrings. Fred Drake <fdrake@acm.org> parents: 17934 diff changeset	161 Call this as often as you want, with as little or as much text
00935031b373 Change some comments into docstrings. Fred Drake <fdrake@acm.org> parents: 17934 diff changeset	162 as you want (may include '\n').
00935031b373 Change some comments into docstrings. Fred Drake <fdrake@acm.org> parents: 17934 diff changeset	163 """
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	164 self.rawdata = self.rawdata + data
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	165 self.goahead(0)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	166
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	167 def close(self):
18984 00935031b373 Change some comments into docstrings. Fred Drake <fdrake@acm.org> parents: 17934 diff changeset	168 """Handle any buffered data."""
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	169 self.goahead(1)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	170
20067 299f5fcb0ba8 Re-factor the HTMLParser class to use the new markupbase.ParserBase class. Fred Drake <fdrake@acm.org> parents: 19953 diff changeset	171 def error(self, message):
86842 0a56709eb798 #15114: The html.parser module now raises a DeprecationWarning when the strict argument of HTMLParser or the HTMLParser.error method are used. Ezio Melotti <ezio.melotti@gmail.com> parents: 83587 diff changeset	172 warnings.warn("The 'error' method is deprecated.",
0a56709eb798 #15114: The html.parser module now raises a DeprecationWarning when the strict argument of HTMLParser or the HTMLParser.error method are used. Ezio Melotti <ezio.melotti@gmail.com> parents: 83587 diff changeset	173 DeprecationWarning, stacklevel=2)
20067 299f5fcb0ba8 Re-factor the HTMLParser class to use the new markupbase.ParserBase class. Fred Drake <fdrake@acm.org> parents: 19953 diff changeset	174 raise HTMLParseError(message, self.getpos())
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	175
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	176 __starttag_text = None
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	177
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	178 def get_starttag_text(self):
18984 00935031b373 Change some comments into docstrings. Fred Drake <fdrake@acm.org> parents: 17934 diff changeset	179 """Return full source of start tag: '<...>'."""
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	180 return self.__starttag_text
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	181
73274 a6f2244b251f #670664: Fix HTMLParser to correctly handle the content of ``<script>...</script>`` and ``<style>...</style>``. Ezio Melotti <ezio.melotti@gmail.com> parents: 73159 diff changeset	182 def set_cdata_mode(self, elem):
a6f2244b251f #670664: Fix HTMLParser to correctly handle the content of ``<script>...</script>`` and ``<style>...</style>``. Ezio Melotti <ezio.melotti@gmail.com> parents: 73159 diff changeset	183 self.cdata_elem = elem.lower()
73609 0a32e7e3aa1f #13358: HTMLParser now calls handle_data only once for each CDATA. Ezio Melotti <ezio.melotti@gmail.com> parents: 73547 diff changeset	184 self.interesting = re.compile(r'</\s%s\s>' % self.cdata_elem, re.I)
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	185
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	186 def clear_cdata_mode(self):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	187 self.interesting = interesting_normal
73274 a6f2244b251f #670664: Fix HTMLParser to correctly handle the content of ``<script>...</script>`` and ``<style>...</style>``. Ezio Melotti <ezio.melotti@gmail.com> parents: 73159 diff changeset	188 self.cdata_elem = None
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	189
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	190 # Internal -- handle data as far as reasonable. May leave state
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	191 # and data to be processed by a subsequent call. If 'end' is
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	192 # true, force handling all data as if followed by EOF marker.
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	193 def goahead(self, end):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	194 rawdata = self.rawdata
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	195 i = 0
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	196 n = len(rawdata)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	197 while i < n:
87437 1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	198 if self.convert_charrefs and not self.cdata_elem:
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	199 j = rawdata.find('<', i)
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	200 if j < 0:
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	201 if not end:
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	202 break # wait till we get all the text
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	203 j = n
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	204 else:
87437 1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	205 match = self.interesting.search(rawdata, i) # < or &
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	206 if match:
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	207 j = match.start()
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	208 else:
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	209 if self.cdata_elem:
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	210 break
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	211 j = n
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	212 if i < j:
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	213 if self.convert_charrefs and not self.cdata_elem:
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	214 self.handle_data(unescape(rawdata[i:j]))
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	215 else:
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	216 self.handle_data(rawdata[i:j])
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	217 i = self.updatepos(i, j)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	218 if i == n: break
21158 1acc4ba73b9d Convert to using string methods instead of the string module. Fred Drake <fdrake@acm.org> parents: 20067 diff changeset	219 startswith = rawdata.startswith
1acc4ba73b9d Convert to using string methods instead of the string module. Fred Drake <fdrake@acm.org> parents: 20067 diff changeset	220 if startswith('<', i):
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	221 if starttagopen.match(rawdata, i): # < + letter
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	222 k = self.parse_starttag(i)
21158 1acc4ba73b9d Convert to using string methods instead of the string module. Fred Drake <fdrake@acm.org> parents: 20067 diff changeset	223 elif startswith("</", i):
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	224 k = self.parse_endtag(i)
21158 1acc4ba73b9d Convert to using string methods instead of the string module. Fred Drake <fdrake@acm.org> parents: 20067 diff changeset	225 elif startswith("<!--", i):
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	226 k = self.parse_comment(i)
21158 1acc4ba73b9d Convert to using string methods instead of the string module. Fred Drake <fdrake@acm.org> parents: 20067 diff changeset	227 elif startswith("<?", i):
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	228 k = self.parse_pi(i)
21158 1acc4ba73b9d Convert to using string methods instead of the string module. Fred Drake <fdrake@acm.org> parents: 20067 diff changeset	229 elif startswith("<!", i):
74859 242b697449d8 #13960: HTMLParser is now able to handle broken comments when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 73609 diff changeset	230 if self.strict:
242b697449d8 #13960: HTMLParser is now able to handle broken comments when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 73609 diff changeset	231 k = self.parse_declaration(i)
242b697449d8 #13960: HTMLParser is now able to handle broken comments when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 73609 diff changeset	232 else:
74906 4c4ff9fd19b6 Improve handling of declarations in HTMLParser. Ezio Melotti <ezio.melotti@gmail.com> parents: 74901 diff changeset	233 k = self.parse_html_declaration(i)
19682 8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases Fred Drake <fdrake@acm.org> parents: 19410 diff changeset	234 elif (i + 1) < n:
19410 c403af003884 Deal more appropriately with bare ampersands and pointy brackets; this Fred Drake <fdrake@acm.org> parents: 18984 diff changeset	235 self.handle_data("<")
c403af003884 Deal more appropriately with bare ampersands and pointy brackets; this Fred Drake <fdrake@acm.org> parents: 18984 diff changeset	236 k = i + 1
19682 8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases Fred Drake <fdrake@acm.org> parents: 19410 diff changeset	237 else:
8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases Fred Drake <fdrake@acm.org> parents: 19410 diff changeset	238 break
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	239 if k < 0:
66623 1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	240 if not end:
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	241 break
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	242 if self.strict:
20067 299f5fcb0ba8 Re-factor the HTMLParser class to use the new markupbase.ParserBase class. Fred Drake <fdrake@acm.org> parents: 19953 diff changeset	243 self.error("EOF in middle of construct")
66623 1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	244 k = rawdata.find('>', i + 1)
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	245 if k < 0:
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	246 k = rawdata.find('<', i + 1)
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	247 if k < 0:
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	248 k = i + 1
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	249 else:
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	250 k += 1
87437 1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	251 if self.convert_charrefs and not self.cdata_elem:
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	252 self.handle_data(unescape(rawdata[i:k]))
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	253 else:
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	254 self.handle_data(rawdata[i:k])
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	255 i = self.updatepos(i, k)
21158 1acc4ba73b9d Convert to using string methods instead of the string module. Fred Drake <fdrake@acm.org> parents: 20067 diff changeset	256 elif startswith("&#", i):
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	257 match = charref.match(rawdata, i)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	258 if match:
18984 00935031b373 Change some comments into docstrings. Fred Drake <fdrake@acm.org> parents: 17934 diff changeset	259 name = match.group()[2:-1]
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	260 self.handle_charref(name)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	261 k = match.end()
21158 1acc4ba73b9d Convert to using string methods instead of the string module. Fred Drake <fdrake@acm.org> parents: 20067 diff changeset	262 if not startswith(';', k-1):
19410 c403af003884 Deal more appropriately with bare ampersands and pointy brackets; this Fred Drake <fdrake@acm.org> parents: 18984 diff changeset	263 k = k - 1
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	264 i = self.updatepos(i, k)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	265 continue
19682 8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases Fred Drake <fdrake@acm.org> parents: 19410 diff changeset	266 else:
61889 bf04eaf1af74 Recorded merge of revisions 81500-81501 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 48769 diff changeset	267 if ";" in rawdata[i:]: #bail by consuming &#
bf04eaf1af74 Recorded merge of revisions 81500-81501 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 48769 diff changeset	268 self.handle_data(rawdata[0:2])
bf04eaf1af74 Recorded merge of revisions 81500-81501 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 48769 diff changeset	269 i = self.updatepos(i, 2)
19682 8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases Fred Drake <fdrake@acm.org> parents: 19410 diff changeset	270 break
21158 1acc4ba73b9d Convert to using string methods instead of the string module. Fred Drake <fdrake@acm.org> parents: 20067 diff changeset	271 elif startswith('&', i):
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	272 match = entityref.match(rawdata, i)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	273 if match:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	274 name = match.group(1)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	275 self.handle_entityref(name)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	276 k = match.end()
21158 1acc4ba73b9d Convert to using string methods instead of the string module. Fred Drake <fdrake@acm.org> parents: 20067 diff changeset	277 if not startswith(';', k-1):
19410 c403af003884 Deal more appropriately with bare ampersands and pointy brackets; this Fred Drake <fdrake@acm.org> parents: 18984 diff changeset	278 k = k - 1
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	279 i = self.updatepos(i, k)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	280 continue
19410 c403af003884 Deal more appropriately with bare ampersands and pointy brackets; this Fred Drake <fdrake@acm.org> parents: 18984 diff changeset	281 match = incomplete.match(rawdata, i)
c403af003884 Deal more appropriately with bare ampersands and pointy brackets; this Fred Drake <fdrake@acm.org> parents: 18984 diff changeset	282 if match:
19682 8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases Fred Drake <fdrake@acm.org> parents: 19410 diff changeset	283 # match.group() will contain at least 2 chars
21158 1acc4ba73b9d Convert to using string methods instead of the string module. Fred Drake <fdrake@acm.org> parents: 20067 diff changeset	284 if end and match.group() == rawdata[i:]:
66623 1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	285 if self.strict:
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	286 self.error("EOF in middle of entity or char ref")
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	287 else:
83586 9cb90c1a1a46 #17802: Fix an UnboundLocalError in html.parser. Initial tests by Thomas Barlow. Ezio Melotti <ezio.melotti@gmail.com> parents: 77742 diff changeset	288 k = match.end()
66623 1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	289 if k <= i:
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	290 k = n
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	291 i = self.updatepos(i, i + 1)
19682 8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases Fred Drake <fdrake@acm.org> parents: 19410 diff changeset	292 # incomplete
8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases Fred Drake <fdrake@acm.org> parents: 19410 diff changeset	293 break
8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases Fred Drake <fdrake@acm.org> parents: 19410 diff changeset	294 elif (i + 1) < n:
8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases Fred Drake <fdrake@acm.org> parents: 19410 diff changeset	295 # not the end of the buffer, and can't be confused
8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases Fred Drake <fdrake@acm.org> parents: 19410 diff changeset	296 # with some other construct
8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases Fred Drake <fdrake@acm.org> parents: 19410 diff changeset	297 self.handle_data("&")
8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases Fred Drake <fdrake@acm.org> parents: 19410 diff changeset	298 i = self.updatepos(i, i + 1)
8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases Fred Drake <fdrake@acm.org> parents: 19410 diff changeset	299 else:
8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases Fred Drake <fdrake@acm.org> parents: 19410 diff changeset	300 break
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	301 else:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	302 assert 0, "interesting.search() lied"
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	303 # end while
73609 0a32e7e3aa1f #13358: HTMLParser now calls handle_data only once for each CDATA. Ezio Melotti <ezio.melotti@gmail.com> parents: 73547 diff changeset	304 if end and i < n and not self.cdata_elem:
87437 1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	305 if self.convert_charrefs and not self.cdata_elem:
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	306 self.handle_data(unescape(rawdata[i:n]))
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	307 else:
1575f2dd08c4 #13633: Added a new convert_charrefs keyword arg to HTMLParser that, when True, automatically converts all character references. Ezio Melotti <ezio.melotti@gmail.com> parents: 87334 diff changeset	308 self.handle_data(rawdata[i:n])
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	309 i = self.updatepos(i, n)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	310 self.rawdata = rawdata[i:]
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	311
74906 4c4ff9fd19b6 Improve handling of declarations in HTMLParser. Ezio Melotti <ezio.melotti@gmail.com> parents: 74901 diff changeset	312 # Internal -- parse html declarations, return length or -1 if not terminated
4c4ff9fd19b6 Improve handling of declarations in HTMLParser. Ezio Melotti <ezio.melotti@gmail.com> parents: 74901 diff changeset	313 # See w3.org/TR/html5/tokenization.html#markup-declaration-open-state
4c4ff9fd19b6 Improve handling of declarations in HTMLParser. Ezio Melotti <ezio.melotti@gmail.com> parents: 74901 diff changeset	314 # See also parse_declaration in _markupbase
4c4ff9fd19b6 Improve handling of declarations in HTMLParser. Ezio Melotti <ezio.melotti@gmail.com> parents: 74901 diff changeset	315 def parse_html_declaration(self, i):
4c4ff9fd19b6 Improve handling of declarations in HTMLParser. Ezio Melotti <ezio.melotti@gmail.com> parents: 74901 diff changeset	316 rawdata = self.rawdata
77630 8dd2f5754b2f #15114: the strict mode of HTMLParser and the HTMLParseError exception are deprecated now that the parser is able to parse invalid markup. Ezio Melotti <ezio.melotti@gmail.com> parents: 76414 diff changeset	317 assert rawdata[i:i+2] == '<!', ('unexpected call to '
8dd2f5754b2f #15114: the strict mode of HTMLParser and the HTMLParseError exception are deprecated now that the parser is able to parse invalid markup. Ezio Melotti <ezio.melotti@gmail.com> parents: 76414 diff changeset	318 'parse_html_declaration()')
74906 4c4ff9fd19b6 Improve handling of declarations in HTMLParser. Ezio Melotti <ezio.melotti@gmail.com> parents: 74901 diff changeset	319 if rawdata[i:i+4] == '<!--':
74913 586d688dcf7f Fix an index and clean up comments. Ezio Melotti <ezio.melotti@gmail.com> parents: 74906 diff changeset	320 # this case is actually already handled in goahead()
74906 4c4ff9fd19b6 Improve handling of declarations in HTMLParser. Ezio Melotti <ezio.melotti@gmail.com> parents: 74901 diff changeset	321 return self.parse_comment(i)
4c4ff9fd19b6 Improve handling of declarations in HTMLParser. Ezio Melotti <ezio.melotti@gmail.com> parents: 74901 diff changeset	322 elif rawdata[i:i+3] == '<![':
4c4ff9fd19b6 Improve handling of declarations in HTMLParser. Ezio Melotti <ezio.melotti@gmail.com> parents: 74901 diff changeset	323 return self.parse_marked_section(i)
4c4ff9fd19b6 Improve handling of declarations in HTMLParser. Ezio Melotti <ezio.melotti@gmail.com> parents: 74901 diff changeset	324 elif rawdata[i:i+9].lower() == '<!doctype':
4c4ff9fd19b6 Improve handling of declarations in HTMLParser. Ezio Melotti <ezio.melotti@gmail.com> parents: 74901 diff changeset	325 # find the closing >
74913 586d688dcf7f Fix an index and clean up comments. Ezio Melotti <ezio.melotti@gmail.com> parents: 74906 diff changeset	326 gtpos = rawdata.find('>', i+9)
74906 4c4ff9fd19b6 Improve handling of declarations in HTMLParser. Ezio Melotti <ezio.melotti@gmail.com> parents: 74901 diff changeset	327 if gtpos == -1:
4c4ff9fd19b6 Improve handling of declarations in HTMLParser. Ezio Melotti <ezio.melotti@gmail.com> parents: 74901 diff changeset	328 return -1
4c4ff9fd19b6 Improve handling of declarations in HTMLParser. Ezio Melotti <ezio.melotti@gmail.com> parents: 74901 diff changeset	329 self.handle_decl(rawdata[i+2:gtpos])
4c4ff9fd19b6 Improve handling of declarations in HTMLParser. Ezio Melotti <ezio.melotti@gmail.com> parents: 74901 diff changeset	330 return gtpos+1
4c4ff9fd19b6 Improve handling of declarations in HTMLParser. Ezio Melotti <ezio.melotti@gmail.com> parents: 74901 diff changeset	331 else:
4c4ff9fd19b6 Improve handling of declarations in HTMLParser. Ezio Melotti <ezio.melotti@gmail.com> parents: 74901 diff changeset	332 return self.parse_bogus_comment(i)
4c4ff9fd19b6 Improve handling of declarations in HTMLParser. Ezio Melotti <ezio.melotti@gmail.com> parents: 74901 diff changeset	333
74859 242b697449d8 #13960: HTMLParser is now able to handle broken comments when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 73609 diff changeset	334 # Internal -- parse bogus comment, return length or -1 if not terminated
242b697449d8 #13960: HTMLParser is now able to handle broken comments when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 73609 diff changeset	335 # see http://www.w3.org/TR/html5/tokenization.html#bogus-comment-state
242b697449d8 #13960: HTMLParser is now able to handle broken comments when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 73609 diff changeset	336 def parse_bogus_comment(self, i, report=1):
242b697449d8 #13960: HTMLParser is now able to handle broken comments when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 73609 diff changeset	337 rawdata = self.rawdata
77630 8dd2f5754b2f #15114: the strict mode of HTMLParser and the HTMLParseError exception are deprecated now that the parser is able to parse invalid markup. Ezio Melotti <ezio.melotti@gmail.com> parents: 76414 diff changeset	338 assert rawdata[i:i+2] in ('<!', '</'), ('unexpected call to '
8dd2f5754b2f #15114: the strict mode of HTMLParser and the HTMLParseError exception are deprecated now that the parser is able to parse invalid markup. Ezio Melotti <ezio.melotti@gmail.com> parents: 76414 diff changeset	339 'parse_comment()')
74859 242b697449d8 #13960: HTMLParser is now able to handle broken comments when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 73609 diff changeset	340 pos = rawdata.find('>', i+2)
242b697449d8 #13960: HTMLParser is now able to handle broken comments when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 73609 diff changeset	341 if pos == -1:
242b697449d8 #13960: HTMLParser is now able to handle broken comments when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 73609 diff changeset	342 return -1
242b697449d8 #13960: HTMLParser is now able to handle broken comments when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 73609 diff changeset	343 if report:
242b697449d8 #13960: HTMLParser is now able to handle broken comments when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 73609 diff changeset	344 self.handle_comment(rawdata[i+2:pos])
242b697449d8 #13960: HTMLParser is now able to handle broken comments when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 73609 diff changeset	345 return pos + 1
242b697449d8 #13960: HTMLParser is now able to handle broken comments when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 73609 diff changeset	346
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	347 # Internal -- parse processing instr, return end or -1 if not terminated
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	348 def parse_pi(self, i):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	349 rawdata = self.rawdata
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	350 assert rawdata[i:i+2] == '<?', 'unexpected call to parse_pi()'
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	351 match = piclose.search(rawdata, i+2) # >
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	352 if not match:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	353 return -1
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	354 j = match.start()
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	355 self.handle_pi(rawdata[i+2: j])
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	356 j = match.end()
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	357 return j
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	358
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	359 # Internal -- handle starttag, return end or -1 if not terminated
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	360 def parse_starttag(self, i):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	361 self.__starttag_text = None
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	362 endpos = self.check_for_whole_start_tag(i)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	363 if endpos < 0:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	364 return endpos
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	365 rawdata = self.rawdata
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	366 self.__starttag_text = rawdata[i:endpos]
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	367
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	368 # Now parse the data between i+1 and j into a tag and attrs
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	369 attrs = []
86989 9b9d188ed549 #19480: HTMLParser now accepts all valid start-tag names as defined by the HTML5 standard. Ezio Melotti <ezio.melotti@gmail.com> parents: 83586 diff changeset	370 if self.strict:
9b9d188ed549 #19480: HTMLParser now accepts all valid start-tag names as defined by the HTML5 standard. Ezio Melotti <ezio.melotti@gmail.com> parents: 83586 diff changeset	371 match = tagfind.match(rawdata, i+1)
9b9d188ed549 #19480: HTMLParser now accepts all valid start-tag names as defined by the HTML5 standard. Ezio Melotti <ezio.melotti@gmail.com> parents: 83586 diff changeset	372 else:
9b9d188ed549 #19480: HTMLParser now accepts all valid start-tag names as defined by the HTML5 standard. Ezio Melotti <ezio.melotti@gmail.com> parents: 83586 diff changeset	373 match = tagfind_tolerant.match(rawdata, i+1)
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	374 assert match, 'unexpected call to parse_starttag()'
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	375 k = match.end()
76414 ba4baaddac8d #14538: HTMLParser can now parse correctly start tags that contain a bare /. Ezio Melotti <ezio.melotti@gmail.com> parents: 75117 diff changeset	376 self.lasttag = tag = match.group(1).lower()
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	377 while k < endpos:
66623 1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	378 if self.strict:
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	379 m = attrfind.match(rawdata, k)
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	380 else:
73159 41d41776aa6d #13273: fix a bug that prevented HTMLParser to properly detect some tags when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 72270 diff changeset	381 m = attrfind_tolerant.match(rawdata, k)
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	382 if not m:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	383 break
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	384 attrname, rest, attrvalue = m.group(1, 2, 3)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	385 if not rest:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	386 attrvalue = None
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	387 elif attrvalue[:1] == '\'' == attrvalue[-1:] or \
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	388 attrvalue[:1] == '"' == attrvalue[-1:]:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	389 attrvalue = attrvalue[1:-1]
73547 16ed15ff0d7c #1745761, #755670, #13357, #12629, #1200313: improve attribute handling in HTMLParser. Ezio Melotti <ezio.melotti@gmail.com> parents: 73274 diff changeset	390 if attrvalue:
87275 7b9235852b3b #2927: Added the unescape() function to the html module. Ezio Melotti <ezio.melotti@gmail.com> parents: 86990 diff changeset	391 attrvalue = unescape(attrvalue)
21158 1acc4ba73b9d Convert to using string methods instead of the string module. Fred Drake <fdrake@acm.org> parents: 20067 diff changeset	392 attrs.append((attrname.lower(), attrvalue))
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	393 k = m.end()
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	394
21158 1acc4ba73b9d Convert to using string methods instead of the string module. Fred Drake <fdrake@acm.org> parents: 20067 diff changeset	395 end = rawdata[k:endpos].strip()
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	396 if end not in (">", "/>"):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	397 lineno, offset = self.getpos()
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	398 if "\n" in self.__starttag_text:
21158 1acc4ba73b9d Convert to using string methods instead of the string module. Fred Drake <fdrake@acm.org> parents: 20067 diff changeset	399 lineno = lineno + self.__starttag_text.count("\n")
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	400 offset = len(self.__starttag_text) \
21158 1acc4ba73b9d Convert to using string methods instead of the string module. Fred Drake <fdrake@acm.org> parents: 20067 diff changeset	401 - self.__starttag_text.rfind("\n")
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	402 else:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	403 offset = offset + len(self.__starttag_text)
66623 1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	404 if self.strict:
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	405 self.error("junk characters in start tag: %r"
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	406 % (rawdata[k:endpos][:20],))
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	407 self.handle_data(rawdata[i:endpos])
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	408 return endpos
21158 1acc4ba73b9d Convert to using string methods instead of the string module. Fred Drake <fdrake@acm.org> parents: 20067 diff changeset	409 if end.endswith('/>'):
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	410 # XHTML-style empty tag: <span attr="value" />
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	411 self.handle_startendtag(tag, attrs)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	412 else:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	413 self.handle_starttag(tag, attrs)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	414 if tag in self.CDATA_CONTENT_ELEMENTS:
73274 a6f2244b251f #670664: Fix HTMLParser to correctly handle the content of ``<script>...</script>`` and ``<style>...</style>``. Ezio Melotti <ezio.melotti@gmail.com> parents: 73159 diff changeset	415 self.set_cdata_mode(tag)
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	416 return endpos
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	417
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	418 # Internal -- check to see if we have a complete starttag; return end
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	419 # or -1 if incomplete.
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	420 def check_for_whole_start_tag(self, i):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	421 rawdata = self.rawdata
66623 1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	422 if self.strict:
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	423 m = locatestarttagend.match(rawdata, i)
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	424 else:
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	425 m = locatestarttagend_tolerant.match(rawdata, i)
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	426 if m:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	427 j = m.end()
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	428 next = rawdata[j:j+1]
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	429 if next == ">":
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	430 return j + 1
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	431 if next == "/":
21158 1acc4ba73b9d Convert to using string methods instead of the string module. Fred Drake <fdrake@acm.org> parents: 20067 diff changeset	432 if rawdata.startswith("/>", j):
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	433 return j + 2
21158 1acc4ba73b9d Convert to using string methods instead of the string module. Fred Drake <fdrake@acm.org> parents: 20067 diff changeset	434 if rawdata.startswith("/", j):
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	435 # buffer boundary
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	436 return -1
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	437 # else bogus input
66623 1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	438 if self.strict:
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	439 self.updatepos(i, j + 1)
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	440 self.error("malformed empty start tag")
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	441 if j > i:
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	442 return j
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	443 else:
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	444 return i + 1
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	445 if next == "":
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	446 # end of input
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	447 return -1
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	448 if next in ("abcdefghijklmnopqrstuvwxyz=/"
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	449 "ABCDEFGHIJKLMNOPQRSTUVWXYZ"):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	450 # end of input in or before attribute value, or we have the
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	451 # '/' from a '/>' ending
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	452 return -1
66623 1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	453 if self.strict:
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	454 self.updatepos(i, j)
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	455 self.error("malformed start tag")
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	456 if j > i:
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	457 return j
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	458 else:
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	459 return i + 1
20067 299f5fcb0ba8 Re-factor the HTMLParser class to use the new markupbase.ParserBase class. Fred Drake <fdrake@acm.org> parents: 19953 diff changeset	460 raise AssertionError("we should not get here!")
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	461
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	462 # Internal -- parse endtag, return end or -1 if incomplete
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	463 def parse_endtag(self, i):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	464 rawdata = self.rawdata
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	465 assert rawdata[i:i+2] == "</", "unexpected call to parse_endtag"
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	466 match = endendtag.search(rawdata, i+1) # >
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	467 if not match:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	468 return -1
74901 df5e5eea7833 #13993: HTMLParser is now able to handle broken end tags when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 74859 diff changeset	469 gtpos = match.end()
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	470 match = endtagfind.match(rawdata, i) # </ + tag + >
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	471 if not match:
73274 a6f2244b251f #670664: Fix HTMLParser to correctly handle the content of ``<script>...</script>`` and ``<style>...</style>``. Ezio Melotti <ezio.melotti@gmail.com> parents: 73159 diff changeset	472 if self.cdata_elem is not None:
74901 df5e5eea7833 #13993: HTMLParser is now able to handle broken end tags when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 74859 diff changeset	473 self.handle_data(rawdata[i:gtpos])
df5e5eea7833 #13993: HTMLParser is now able to handle broken end tags when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 74859 diff changeset	474 return gtpos
66623 1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	475 if self.strict:
74901 df5e5eea7833 #13993: HTMLParser is now able to handle broken end tags when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 74859 diff changeset	476 self.error("bad end tag: %r" % (rawdata[i:gtpos],))
df5e5eea7833 #13993: HTMLParser is now able to handle broken end tags when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 74859 diff changeset	477 # find the name: w3.org/TR/html5/tokenization.html#tag-name-state
df5e5eea7833 #13993: HTMLParser is now able to handle broken end tags when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 74859 diff changeset	478 namematch = tagfind_tolerant.match(rawdata, i+2)
df5e5eea7833 #13993: HTMLParser is now able to handle broken end tags when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 74859 diff changeset	479 if not namematch:
df5e5eea7833 #13993: HTMLParser is now able to handle broken end tags when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 74859 diff changeset	480 # w3.org/TR/html5/tokenization.html#end-tag-open-state
df5e5eea7833 #13993: HTMLParser is now able to handle broken end tags when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 74859 diff changeset	481 if rawdata[i:i+3] == '</>':
df5e5eea7833 #13993: HTMLParser is now able to handle broken end tags when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 74859 diff changeset	482 return i+3
df5e5eea7833 #13993: HTMLParser is now able to handle broken end tags when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 74859 diff changeset	483 else:
df5e5eea7833 #13993: HTMLParser is now able to handle broken end tags when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 74859 diff changeset	484 return self.parse_bogus_comment(i)
86989 9b9d188ed549 #19480: HTMLParser now accepts all valid start-tag names as defined by the HTML5 standard. Ezio Melotti <ezio.melotti@gmail.com> parents: 83586 diff changeset	485 tagname = namematch.group(1).lower()
74901 df5e5eea7833 #13993: HTMLParser is now able to handle broken end tags when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 74859 diff changeset	486 # consume and ignore other stuff between the name and the >
df5e5eea7833 #13993: HTMLParser is now able to handle broken end tags when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 74859 diff changeset	487 # Note: this is not 100% correct, since we might have things like
df5e5eea7833 #13993: HTMLParser is now able to handle broken end tags when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 74859 diff changeset	488 # </tag attr=">">, but looking for > after tha name should cover
df5e5eea7833 #13993: HTMLParser is now able to handle broken end tags when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 74859 diff changeset	489 # most of the cases and is much simpler
df5e5eea7833 #13993: HTMLParser is now able to handle broken end tags when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 74859 diff changeset	490 gtpos = rawdata.find('>', namematch.end())
df5e5eea7833 #13993: HTMLParser is now able to handle broken end tags when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 74859 diff changeset	491 self.handle_endtag(tagname)
df5e5eea7833 #13993: HTMLParser is now able to handle broken end tags when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 74859 diff changeset	492 return gtpos+1
73274 a6f2244b251f #670664: Fix HTMLParser to correctly handle the content of ``<script>...</script>`` and ``<style>...</style>``. Ezio Melotti <ezio.melotti@gmail.com> parents: 73159 diff changeset	493
a6f2244b251f #670664: Fix HTMLParser to correctly handle the content of ``<script>...</script>`` and ``<style>...</style>``. Ezio Melotti <ezio.melotti@gmail.com> parents: 73159 diff changeset	494 elem = match.group(1).lower() # script or style
a6f2244b251f #670664: Fix HTMLParser to correctly handle the content of ``<script>...</script>`` and ``<style>...</style>``. Ezio Melotti <ezio.melotti@gmail.com> parents: 73159 diff changeset	495 if self.cdata_elem is not None:
a6f2244b251f #670664: Fix HTMLParser to correctly handle the content of ``<script>...</script>`` and ``<style>...</style>``. Ezio Melotti <ezio.melotti@gmail.com> parents: 73159 diff changeset	496 if elem != self.cdata_elem:
74901 df5e5eea7833 #13993: HTMLParser is now able to handle broken end tags when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 74859 diff changeset	497 self.handle_data(rawdata[i:gtpos])
df5e5eea7833 #13993: HTMLParser is now able to handle broken end tags when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 74859 diff changeset	498 return gtpos
73274 a6f2244b251f #670664: Fix HTMLParser to correctly handle the content of ``<script>...</script>`` and ``<style>...</style>``. Ezio Melotti <ezio.melotti@gmail.com> parents: 73159 diff changeset	499
a6f2244b251f #670664: Fix HTMLParser to correctly handle the content of ``<script>...</script>`` and ``<style>...</style>``. Ezio Melotti <ezio.melotti@gmail.com> parents: 73159 diff changeset	500 self.handle_endtag(elem.lower())
23452 6bae0c31bbfb Simplify code to remove an unnecessary test. Fred Drake <fdrake@acm.org> parents: 21158 diff changeset	501 self.clear_cdata_mode()
74901 df5e5eea7833 #13993: HTMLParser is now able to handle broken end tags when strict=False. Ezio Melotti <ezio.melotti@gmail.com> parents: 74859 diff changeset	502 return gtpos
17853 055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	503
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	504 # Overridable -- finish processing of start+end tag: <tag.../>
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	505 def handle_startendtag(self, tag, attrs):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	506 self.handle_starttag(tag, attrs)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	507 self.handle_endtag(tag)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	508
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	509 # Overridable -- handle start tag
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	510 def handle_starttag(self, tag, attrs):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	511 pass
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	512
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	513 # Overridable -- handle end tag
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	514 def handle_endtag(self, tag):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	515 pass
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	516
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	517 # Overridable -- handle character reference
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	518 def handle_charref(self, name):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	519 pass
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	520
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	521 # Overridable -- handle entity reference
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	522 def handle_entityref(self, name):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	523 pass
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	524
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	525 # Overridable -- handle data
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	526 def handle_data(self, data):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	527 pass
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	528
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	529 # Overridable -- handle comment
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	530 def handle_comment(self, data):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	531 pass
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	532
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	533 # Overridable -- handle declaration
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	534 def handle_decl(self, decl):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	535 pass
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	536
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	537 # Overridable -- handle processing instruction
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	538 def handle_pi(self, data):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	539 pass
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is Guido van Rossum <guido@python.org> parents: diff changeset	540
20067 299f5fcb0ba8 Re-factor the HTMLParser class to use the new markupbase.ParserBase class. Fred Drake <fdrake@acm.org> parents: 19953 diff changeset	541 def unknown_decl(self, data):
66623 1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	542 if self.strict:
1cbfeffea19f #1486713: Add a tolerant mode to HTMLParser. R. David Murray <rdmurray@bitdance.com> parents: 61889 diff changeset	543 self.error("unknown declaration: %r" % (data,))
87334 0534b882f9ce #19688: add back and deprecate the internal HTMLParser.unescape() method. Ezio Melotti <ezio.melotti@gmail.com> parents: 87275 diff changeset	544
0534b882f9ce #19688: add back and deprecate the internal HTMLParser.unescape() method. Ezio Melotti <ezio.melotti@gmail.com> parents: 87275 diff changeset	545 # Internal -- helper to remove special character quoting
0534b882f9ce #19688: add back and deprecate the internal HTMLParser.unescape() method. Ezio Melotti <ezio.melotti@gmail.com> parents: 87275 diff changeset	546 def unescape(self, s):
0534b882f9ce #19688: add back and deprecate the internal HTMLParser.unescape() method. Ezio Melotti <ezio.melotti@gmail.com> parents: 87275 diff changeset	547 warnings.warn('The unescape method is deprecated and will be removed '
0534b882f9ce #19688: add back and deprecate the internal HTMLParser.unescape() method. Ezio Melotti <ezio.melotti@gmail.com> parents: 87275 diff changeset	548 'in 3.5, use html.unescape() instead.',
0534b882f9ce #19688: add back and deprecate the internal HTMLParser.unescape() method. Ezio Melotti <ezio.melotti@gmail.com> parents: 87275 diff changeset	549 DeprecationWarning, stacklevel=2)
0534b882f9ce #19688: add back and deprecate the internal HTMLParser.unescape() method. Ezio Melotti <ezio.melotti@gmail.com> parents: 87275 diff changeset	550 return unescape(s)

Mercurial > cpython-lru_cache-weakref

annotate Lib/html/parser.py @ 87690:66c1c9f32567 default tip