Mercurial > cpython-lru_cache-weakref

annotate Lib/html/parser.py @ 66978:aed2bbafa550 3.1

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Blocked revisions 87324 via svnmerge ........ r87324 | r.david.murray | 2010-12-16 14:08:51 -0500 (Thu, 16 Dec 2010) | 9 lines #10719: restore messages generated on invalid compileall args Before the introduction of filename arguments to compileall it gave semi useful messages about not being able to 'list' names that weren't valid directories. This fix restores that behavior. In addition to the test for this case, the patch also adds a test for the default behavior of compileall when no arguments are provided, and fixes a bug in one of the previously added tests. ........
author R. David Murray <rdmurray@bitdance.com>
date Thu, 16 Dec 2010 19:10:16 +0000
parents 6144760e8350
children 3546ce1aa355 1cbfeffea19f
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
18984
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
1 """A parser for HTML and XHTML."""
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
2
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
3 # This file is based on sgmllib.py, but the API is slightly different.
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
4
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
5 # XXX There should be a way to distinguish between PCDATA (parsed
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
6 # character data -- the normal case), RCDATA (replaceable character
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
7 # data -- only char and entity references and end tags are special)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
8 # and CDATA (character data -- only end tags are special).
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
9
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
10
44212
c6fb393b6c81 rename markupbase to _markupbase
Fred Drake <fdrake@acm.org>
parents: 41585
diff changeset
11 import _markupbase
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
12 import re
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
13
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
14 # Regular expressions used for parsing
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
15
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
16 interesting_normal = re.compile('[&<]')
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
17 interesting_cdata = re.compile(r'<(/|\Z)')
19682
8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases
Fred Drake <fdrake@acm.org>
parents: 19410
diff changeset
18 incomplete = re.compile('&[a-zA-Z#]')
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
19
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
20 entityref = re.compile('&([a-zA-Z][-.a-zA-Z0-9]*)[^a-zA-Z0-9]')
18984
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
21 charref = re.compile('&#(?:[0-9]+|[xX][0-9a-fA-F]+)[^0-9a-fA-F]')
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
22
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
23 starttagopen = re.compile('<[a-zA-Z]')
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
24 piclose = re.compile('>')
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
25 commentclose = re.compile(r'--\s*>')
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
26 tagfind = re.compile('[a-zA-Z][-.a-zA-Z0-9:_]*')
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
27 attrfind = re.compile(
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
28 r'\s*([a-zA-Z_][-.:a-zA-Z_0-9]*)(\s*=\s*'
32256
2f5f38f4bc2b [Bug #921657] Allow '@' in unquoted HTML attributes. Not strictly legal according to the HTML REC, but HTMLParser is already a pretty loose parser. Reported by Bernd Zimmermann.
Andrew M. Kuchling <amk@amk.ca>
parents: 31515
diff changeset
29 r'(\'[^\']*\'|"[^"]*"|[-a-zA-Z0-9./,:;+*%?!&$\(\)_#=~@]*))?')
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
30
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
31 locatestarttagend = re.compile(r"""
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
32 <[a-zA-Z][-.a-zA-Z0-9:_]* # tag name
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
33 (?:\s+ # whitespace before attribute name
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
34 (?:[a-zA-Z_][-.:a-zA-Z0-9_]* # attribute name
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
35 (?:\s*=\s* # value indicator
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
36 (?:'[^']*' # LITA-enclosed value
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
37 |\"[^\"]*\" # LIT-enclosed value
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
38 |[^'\">\s]+ # bare value
35507
15a8975f7fa9 Reverting previous checkin. This breaks too much of HTMLParser to be applied
Georg Brandl <georg@python.org>
parents: 35499
diff changeset
39 )
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
40 )?
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
41 )
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
42 )*
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
43 \s* # trailing whitespace
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
44 """, re.VERBOSE)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
45 endendtag = re.compile('>')
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
46 endtagfind = re.compile('</\s*([a-zA-Z][-.a-zA-Z0-9:_]*)\s*>')
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
47
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
48
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
49 class HTMLParseError(Exception):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
50 """Exception raised for all parse errors."""
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
51
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
52 def __init__(self, msg, position=(None, None)):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
53 assert msg
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
54 self.msg = msg
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
55 self.lineno = position[0]
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
56 self.offset = position[1]
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
57
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
58 def __str__(self):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
59 result = self.msg
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
60 if self.lineno is not None:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
61 result = result + ", at line %d" % self.lineno
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
62 if self.offset is not None:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
63 result = result + ", column %d" % (self.offset + 1)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
64 return result
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
65
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
66
44212
c6fb393b6c81 rename markupbase to _markupbase
Fred Drake <fdrake@acm.org>
parents: 41585
diff changeset
67 class HTMLParser(_markupbase.ParserBase):
18984
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
68 """Find tags and other markup and call handler functions.
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
69
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
70 Usage:
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
71 p = HTMLParser()
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
72 p.feed(data)
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
73 ...
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
74 p.close()
17932
ce7f0e380830 Removed incorrect comment left over from sgmllib.py.
Guido van Rossum <guido@python.org>
parents: 17853
diff changeset
75
18984
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
76 Start tags are handled by calling self.handle_starttag() or
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
77 self.handle_startendtag(); end tags by self.handle_endtag(). The
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
78 data between tags is passed from the parser to the derived class
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
79 by calling self.handle_data() with the data as argument (the data
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
80 may be split up in arbitrary chunks). Entity references are
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
81 passed by calling self.handle_entityref() with the entity
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
82 reference as the argument. Numeric character references are
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
83 passed to self.handle_charref() with the string containing the
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
84 reference as the argument.
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
85 """
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
86
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
87 CDATA_CONTENT_ELEMENTS = ("script", "style")
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
88
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
89
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
90 def __init__(self):
18984
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
91 """Initialize and reset this instance."""
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
92 self.reset()
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
93
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
94 def reset(self):
18984
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
95 """Reset this instance. Loses all unprocessed data."""
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
96 self.rawdata = ''
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
97 self.lasttag = '???'
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
98 self.interesting = interesting_normal
44212
c6fb393b6c81 rename markupbase to _markupbase
Fred Drake <fdrake@acm.org>
parents: 41585
diff changeset
99 _markupbase.ParserBase.reset(self)
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
100
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
101 def feed(self, data):
18984
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
102 """Feed data to the parser.
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
103
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
104 Call this as often as you want, with as little or as much text
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
105 as you want (may include '\n').
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
106 """
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
107 self.rawdata = self.rawdata + data
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
108 self.goahead(0)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
109
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
110 def close(self):
18984
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
111 """Handle any buffered data."""
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
112 self.goahead(1)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
113
20067
299f5fcb0ba8 Re-factor the HTMLParser class to use the new markupbase.ParserBase class.
Fred Drake <fdrake@acm.org>
parents: 19953
diff changeset
114 def error(self, message):
299f5fcb0ba8 Re-factor the HTMLParser class to use the new markupbase.ParserBase class.
Fred Drake <fdrake@acm.org>
parents: 19953
diff changeset
115 raise HTMLParseError(message, self.getpos())
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
116
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
117 __starttag_text = None
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
118
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
119 def get_starttag_text(self):
18984
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
120 """Return full source of start tag: '<...>'."""
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
121 return self.__starttag_text
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
122
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
123 def set_cdata_mode(self):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
124 self.interesting = interesting_cdata
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
125
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
126 def clear_cdata_mode(self):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
127 self.interesting = interesting_normal
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
128
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
129 # Internal -- handle data as far as reasonable. May leave state
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
130 # and data to be processed by a subsequent call. If 'end' is
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
131 # true, force handling all data as if followed by EOF marker.
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
132 def goahead(self, end):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
133 rawdata = self.rawdata
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
134 i = 0
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
135 n = len(rawdata)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
136 while i < n:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
137 match = self.interesting.search(rawdata, i) # < or &
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
138 if match:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
139 j = match.start()
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
140 else:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
141 j = n
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
142 if i < j: self.handle_data(rawdata[i:j])
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
143 i = self.updatepos(i, j)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
144 if i == n: break
21158
1acc4ba73b9d Convert to using string methods instead of the string module.
Fred Drake <fdrake@acm.org>
parents: 20067
diff changeset
145 startswith = rawdata.startswith
1acc4ba73b9d Convert to using string methods instead of the string module.
Fred Drake <fdrake@acm.org>
parents: 20067
diff changeset
146 if startswith('<', i):
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
147 if starttagopen.match(rawdata, i): # < + letter
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
148 k = self.parse_starttag(i)
21158
1acc4ba73b9d Convert to using string methods instead of the string module.
Fred Drake <fdrake@acm.org>
parents: 20067
diff changeset
149 elif startswith("</", i):
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
150 k = self.parse_endtag(i)
21158
1acc4ba73b9d Convert to using string methods instead of the string module.
Fred Drake <fdrake@acm.org>
parents: 20067
diff changeset
151 elif startswith("<!--", i):
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
152 k = self.parse_comment(i)
21158
1acc4ba73b9d Convert to using string methods instead of the string module.
Fred Drake <fdrake@acm.org>
parents: 20067
diff changeset
153 elif startswith("<?", i):
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
154 k = self.parse_pi(i)
21158
1acc4ba73b9d Convert to using string methods instead of the string module.
Fred Drake <fdrake@acm.org>
parents: 20067
diff changeset
155 elif startswith("<!", i):
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
156 k = self.parse_declaration(i)
19682
8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases
Fred Drake <fdrake@acm.org>
parents: 19410
diff changeset
157 elif (i + 1) < n:
19410
c403af003884 Deal more appropriately with bare ampersands and pointy brackets; this
Fred Drake <fdrake@acm.org>
parents: 18984
diff changeset
158 self.handle_data("<")
c403af003884 Deal more appropriately with bare ampersands and pointy brackets; this
Fred Drake <fdrake@acm.org>
parents: 18984
diff changeset
159 k = i + 1
19682
8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases
Fred Drake <fdrake@acm.org>
parents: 19410
diff changeset
160 else:
8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases
Fred Drake <fdrake@acm.org>
parents: 19410
diff changeset
161 break
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
162 if k < 0:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
163 if end:
20067
299f5fcb0ba8 Re-factor the HTMLParser class to use the new markupbase.ParserBase class.
Fred Drake <fdrake@acm.org>
parents: 19953
diff changeset
164 self.error("EOF in middle of construct")
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
165 break
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
166 i = self.updatepos(i, k)
21158
1acc4ba73b9d Convert to using string methods instead of the string module.
Fred Drake <fdrake@acm.org>
parents: 20067
diff changeset
167 elif startswith("&#", i):
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
168 match = charref.match(rawdata, i)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
169 if match:
18984
00935031b373 Change some comments into docstrings.
Fred Drake <fdrake@acm.org>
parents: 17934
diff changeset
170 name = match.group()[2:-1]
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
171 self.handle_charref(name)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
172 k = match.end()
21158
1acc4ba73b9d Convert to using string methods instead of the string module.
Fred Drake <fdrake@acm.org>
parents: 20067
diff changeset
173 if not startswith(';', k-1):
19410
c403af003884 Deal more appropriately with bare ampersands and pointy brackets; this
Fred Drake <fdrake@acm.org>
parents: 18984
diff changeset
174 k = k - 1
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
175 i = self.updatepos(i, k)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
176 continue
19682
8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases
Fred Drake <fdrake@acm.org>
parents: 19410
diff changeset
177 else:
61890
6144760e8350 Merged revisions 81504 via svnmerge from
Victor Stinner <victor.stinner@haypocalc.com>
parents: 48769
diff changeset
178 if ";" in rawdata[i:]: #bail by consuming &#
6144760e8350 Merged revisions 81504 via svnmerge from
Victor Stinner <victor.stinner@haypocalc.com>
parents: 48769
diff changeset
179 self.handle_data(rawdata[0:2])
6144760e8350 Merged revisions 81504 via svnmerge from
Victor Stinner <victor.stinner@haypocalc.com>
parents: 48769
diff changeset
180 i = self.updatepos(i, 2)
19682
8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases
Fred Drake <fdrake@acm.org>
parents: 19410
diff changeset
181 break
21158
1acc4ba73b9d Convert to using string methods instead of the string module.
Fred Drake <fdrake@acm.org>
parents: 20067
diff changeset
182 elif startswith('&', i):
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
183 match = entityref.match(rawdata, i)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
184 if match:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
185 name = match.group(1)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
186 self.handle_entityref(name)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
187 k = match.end()
21158
1acc4ba73b9d Convert to using string methods instead of the string module.
Fred Drake <fdrake@acm.org>
parents: 20067
diff changeset
188 if not startswith(';', k-1):
19410
c403af003884 Deal more appropriately with bare ampersands and pointy brackets; this
Fred Drake <fdrake@acm.org>
parents: 18984
diff changeset
189 k = k - 1
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
190 i = self.updatepos(i, k)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
191 continue
19410
c403af003884 Deal more appropriately with bare ampersands and pointy brackets; this
Fred Drake <fdrake@acm.org>
parents: 18984
diff changeset
192 match = incomplete.match(rawdata, i)
c403af003884 Deal more appropriately with bare ampersands and pointy brackets; this
Fred Drake <fdrake@acm.org>
parents: 18984
diff changeset
193 if match:
19682
8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases
Fred Drake <fdrake@acm.org>
parents: 19410
diff changeset
194 # match.group() will contain at least 2 chars
21158
1acc4ba73b9d Convert to using string methods instead of the string module.
Fred Drake <fdrake@acm.org>
parents: 20067
diff changeset
195 if end and match.group() == rawdata[i:]:
20067
299f5fcb0ba8 Re-factor the HTMLParser class to use the new markupbase.ParserBase class.
Fred Drake <fdrake@acm.org>
parents: 19953
diff changeset
196 self.error("EOF in middle of entity or char ref")
19682
8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases
Fred Drake <fdrake@acm.org>
parents: 19410
diff changeset
197 # incomplete
8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases
Fred Drake <fdrake@acm.org>
parents: 19410
diff changeset
198 break
8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases
Fred Drake <fdrake@acm.org>
parents: 19410
diff changeset
199 elif (i + 1) < n:
8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases
Fred Drake <fdrake@acm.org>
parents: 19410
diff changeset
200 # not the end of the buffer, and can't be confused
8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases
Fred Drake <fdrake@acm.org>
parents: 19410
diff changeset
201 # with some other construct
8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases
Fred Drake <fdrake@acm.org>
parents: 19410
diff changeset
202 self.handle_data("&")
8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases
Fred Drake <fdrake@acm.org>
parents: 19410
diff changeset
203 i = self.updatepos(i, i + 1)
8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases
Fred Drake <fdrake@acm.org>
parents: 19410
diff changeset
204 else:
8dfb13a4e175 Added reasonable parsing of the DOCTYPE declaration, fixed edge cases
Fred Drake <fdrake@acm.org>
parents: 19410
diff changeset
205 break
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
206 else:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
207 assert 0, "interesting.search() lied"
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
208 # end while
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
209 if end and i < n:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
210 self.handle_data(rawdata[i:n])
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
211 i = self.updatepos(i, n)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
212 self.rawdata = rawdata[i:]
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
213
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
214 # Internal -- parse processing instr, return end or -1 if not terminated
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
215 def parse_pi(self, i):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
216 rawdata = self.rawdata
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
217 assert rawdata[i:i+2] == '<?', 'unexpected call to parse_pi()'
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
218 match = piclose.search(rawdata, i+2) # >
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
219 if not match:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
220 return -1
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
221 j = match.start()
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
222 self.handle_pi(rawdata[i+2: j])
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
223 j = match.end()
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
224 return j
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
225
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
226 # Internal -- handle starttag, return end or -1 if not terminated
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
227 def parse_starttag(self, i):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
228 self.__starttag_text = None
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
229 endpos = self.check_for_whole_start_tag(i)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
230 if endpos < 0:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
231 return endpos
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
232 rawdata = self.rawdata
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
233 self.__starttag_text = rawdata[i:endpos]
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
234
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
235 # Now parse the data between i+1 and j into a tag and attrs
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
236 attrs = []
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
237 match = tagfind.match(rawdata, i+1)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
238 assert match, 'unexpected call to parse_starttag()'
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
239 k = match.end()
21158
1acc4ba73b9d Convert to using string methods instead of the string module.
Fred Drake <fdrake@acm.org>
parents: 20067
diff changeset
240 self.lasttag = tag = rawdata[i+1:k].lower()
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
241
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
242 while k < endpos:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
243 m = attrfind.match(rawdata, k)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
244 if not m:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
245 break
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
246 attrname, rest, attrvalue = m.group(1, 2, 3)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
247 if not rest:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
248 attrvalue = None
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
249 elif attrvalue[:1] == '\'' == attrvalue[-1:] or \
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
250 attrvalue[:1] == '"' == attrvalue[-1:]:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
251 attrvalue = attrvalue[1:-1]
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
252 attrvalue = self.unescape(attrvalue)
21158
1acc4ba73b9d Convert to using string methods instead of the string module.
Fred Drake <fdrake@acm.org>
parents: 20067
diff changeset
253 attrs.append((attrname.lower(), attrvalue))
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
254 k = m.end()
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
255
21158
1acc4ba73b9d Convert to using string methods instead of the string module.
Fred Drake <fdrake@acm.org>
parents: 20067
diff changeset
256 end = rawdata[k:endpos].strip()
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
257 if end not in (">", "/>"):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
258 lineno, offset = self.getpos()
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
259 if "\n" in self.__starttag_text:
21158
1acc4ba73b9d Convert to using string methods instead of the string module.
Fred Drake <fdrake@acm.org>
parents: 20067
diff changeset
260 lineno = lineno + self.__starttag_text.count("\n")
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
261 offset = len(self.__starttag_text) \
21158
1acc4ba73b9d Convert to using string methods instead of the string module.
Fred Drake <fdrake@acm.org>
parents: 20067
diff changeset
262 - self.__starttag_text.rfind("\n")
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
263 else:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
264 offset = offset + len(self.__starttag_text)
31515
71a294bd7950 Replace backticks with repr() or "%r"
Walter Dörwald <walter@livinglogic.de>
parents: 28297
diff changeset
265 self.error("junk characters in start tag: %r"
71a294bd7950 Replace backticks with repr() or "%r"
Walter Dörwald <walter@livinglogic.de>
parents: 28297
diff changeset
266 % (rawdata[k:endpos][:20],))
21158
1acc4ba73b9d Convert to using string methods instead of the string module.
Fred Drake <fdrake@acm.org>
parents: 20067
diff changeset
267 if end.endswith('/>'):
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
268 # XHTML-style empty tag: <span attr="value" />
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
269 self.handle_startendtag(tag, attrs)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
270 else:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
271 self.handle_starttag(tag, attrs)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
272 if tag in self.CDATA_CONTENT_ELEMENTS:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
273 self.set_cdata_mode()
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
274 return endpos
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
275
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
276 # Internal -- check to see if we have a complete starttag; return end
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
277 # or -1 if incomplete.
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
278 def check_for_whole_start_tag(self, i):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
279 rawdata = self.rawdata
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
280 m = locatestarttagend.match(rawdata, i)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
281 if m:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
282 j = m.end()
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
283 next = rawdata[j:j+1]
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
284 if next == ">":
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
285 return j + 1
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
286 if next == "/":
21158
1acc4ba73b9d Convert to using string methods instead of the string module.
Fred Drake <fdrake@acm.org>
parents: 20067
diff changeset
287 if rawdata.startswith("/>", j):
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
288 return j + 2
21158
1acc4ba73b9d Convert to using string methods instead of the string module.
Fred Drake <fdrake@acm.org>
parents: 20067
diff changeset
289 if rawdata.startswith("/", j):
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
290 # buffer boundary
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
291 return -1
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
292 # else bogus input
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
293 self.updatepos(i, j + 1)
20067
299f5fcb0ba8 Re-factor the HTMLParser class to use the new markupbase.ParserBase class.
Fred Drake <fdrake@acm.org>
parents: 19953
diff changeset
294 self.error("malformed empty start tag")
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
295 if next == "":
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
296 # end of input
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
297 return -1
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
298 if next in ("abcdefghijklmnopqrstuvwxyz=/"
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
299 "ABCDEFGHIJKLMNOPQRSTUVWXYZ"):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
300 # end of input in or before attribute value, or we have the
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
301 # '/' from a '/>' ending
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
302 return -1
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
303 self.updatepos(i, j)
20067
299f5fcb0ba8 Re-factor the HTMLParser class to use the new markupbase.ParserBase class.
Fred Drake <fdrake@acm.org>
parents: 19953
diff changeset
304 self.error("malformed start tag")
299f5fcb0ba8 Re-factor the HTMLParser class to use the new markupbase.ParserBase class.
Fred Drake <fdrake@acm.org>
parents: 19953
diff changeset
305 raise AssertionError("we should not get here!")
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
306
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
307 # Internal -- parse endtag, return end or -1 if incomplete
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
308 def parse_endtag(self, i):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
309 rawdata = self.rawdata
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
310 assert rawdata[i:i+2] == "</", "unexpected call to parse_endtag"
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
311 match = endendtag.search(rawdata, i+1) # >
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
312 if not match:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
313 return -1
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
314 j = match.end()
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
315 match = endtagfind.match(rawdata, i) # </ + tag + >
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
316 if not match:
31515
71a294bd7950 Replace backticks with repr() or "%r"
Walter Dörwald <walter@livinglogic.de>
parents: 28297
diff changeset
317 self.error("bad end tag: %r" % (rawdata[i:j],))
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
318 tag = match.group(1)
21158
1acc4ba73b9d Convert to using string methods instead of the string module.
Fred Drake <fdrake@acm.org>
parents: 20067
diff changeset
319 self.handle_endtag(tag.lower())
23452
6bae0c31bbfb Simplify code to remove an unnecessary test.
Fred Drake <fdrake@acm.org>
parents: 21158
diff changeset
320 self.clear_cdata_mode()
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
321 return j
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
322
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
323 # Overridable -- finish processing of start+end tag: <tag.../>
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
324 def handle_startendtag(self, tag, attrs):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
325 self.handle_starttag(tag, attrs)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
326 self.handle_endtag(tag)
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
327
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
328 # Overridable -- handle start tag
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
329 def handle_starttag(self, tag, attrs):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
330 pass
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
331
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
332 # Overridable -- handle end tag
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
333 def handle_endtag(self, tag):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
334 pass
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
335
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
336 # Overridable -- handle character reference
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
337 def handle_charref(self, name):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
338 pass
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
339
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
340 # Overridable -- handle entity reference
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
341 def handle_entityref(self, name):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
342 pass
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
343
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
344 # Overridable -- handle data
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
345 def handle_data(self, data):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
346 pass
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
347
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
348 # Overridable -- handle comment
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
349 def handle_comment(self, data):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
350 pass
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
351
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
352 # Overridable -- handle declaration
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
353 def handle_decl(self, decl):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
354 pass
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
355
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
356 # Overridable -- handle processing instruction
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
357 def handle_pi(self, data):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
358 pass
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
359
20067
299f5fcb0ba8 Re-factor the HTMLParser class to use the new markupbase.ParserBase class.
Fred Drake <fdrake@acm.org>
parents: 19953
diff changeset
360 def unknown_decl(self, data):
31515
71a294bd7950 Replace backticks with repr() or "%r"
Walter Dörwald <walter@livinglogic.de>
parents: 28297
diff changeset
361 self.error("unknown declaration: %r" % (data,))
20067
299f5fcb0ba8 Re-factor the HTMLParser class to use the new markupbase.ParserBase class.
Fred Drake <fdrake@acm.org>
parents: 19953
diff changeset
362
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
363 # Internal -- helper to remove special character quoting
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 35507
diff changeset
364 entitydefs = None
17853
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
365 def unescape(self, s):
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
366 if '&' not in s:
055fc046e080 A much improved HTML parser -- a replacement for sgmllib. The API is
Guido van Rossum <guido@python.org>
parents:
diff changeset
367 return s
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 35507
diff changeset
368 def replaceEntities(s):
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 35507
diff changeset
369 s = s.groups()[0]
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 35507
diff changeset
370 if s[0] == "#":
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 35507
diff changeset
371 s = s[1:]
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 35507
diff changeset
372 if s[0] in ['x','X']:
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 35507
diff changeset
373 c = int(s[1:], 16)
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 35507
diff changeset
374 else:
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 35507
diff changeset
375 c = int(s)
41585
825d966338ea Rename 'unicode' to 'str' in its tp_name field. Rename 'str' to 'str8'.
Guido van Rossum <guido@python.org>
parents: 41575
diff changeset
376 return chr(c)
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 35507
diff changeset
377 else:
47147
7296a2fb4786 rename HTMLParser to html.parser and htmlentitydefs to html.entities;
Fred Drake <fdrake@acm.org>
parents: 44212
diff changeset
378 # Cannot use name2codepoint directly, because HTMLParser
7296a2fb4786 rename HTMLParser to html.parser and htmlentitydefs to html.entities;
Fred Drake <fdrake@acm.org>
parents: 44212
diff changeset
379 # supports apos, which is not part of HTML 4
7296a2fb4786 rename HTMLParser to html.parser and htmlentitydefs to html.entities;
Fred Drake <fdrake@acm.org>
parents: 44212
diff changeset
380 import html.entities
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 35507
diff changeset
381 if HTMLParser.entitydefs is None:
41575
beacf40b5960 Rip out all the u"..." literals and calls to unicode().
Guido van Rossum <guido@python.org>
parents: 41554
diff changeset
382 entitydefs = HTMLParser.entitydefs = {'apos':"'"}
47147
7296a2fb4786 rename HTMLParser to html.parser and htmlentitydefs to html.entities;
Fred Drake <fdrake@acm.org>
parents: 44212
diff changeset
383 for k, v in html.entities.name2codepoint.items():
47205
7aa9ad658014 Change test_htmlparser to reflect the HTMLParser -> html.parser
Mark Dickinson <dickinsm@gmail.com>
parents: 47147
diff changeset
384 entitydefs[k] = chr(v)
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 35507
diff changeset
385 try:
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 35507
diff changeset
386 return self.entitydefs[s]
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 35507
diff changeset
387 except KeyError:
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 35507
diff changeset
388 return '&'+s+';'
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 35507
diff changeset
389
47147
7296a2fb4786 rename HTMLParser to html.parser and htmlentitydefs to html.entities;
Fred Drake <fdrake@acm.org>
parents: 44212
diff changeset
390 return re.sub(r"&(#?[xX]?(?:[0-9a-fA-F]+|\w{1,8}));",
48769
83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden,
Antoine Pitrou <solipsis@pitrou.net>
parents: 47205
diff changeset
391 replaceEntities, s, re.ASCII)