Mercurial > cpython-lru_cache-weakref
annotate Tools/webchecker/webchecker.py @ 5817:48b94de85bf3 legacy-trunk
Extensive changes by AMK.
| author | Guido van Rossum <guido@python.org> |
|---|---|
| date | Thu, 17 Jul 1997 16:29:42 +0000 |
| parents | bccb8d742964 |
| children | 9ed9eb4a7f23 |
| rev | line source |
|---|---|
| 5062 | 1 #! /usr/bin/env python |
| 2 | |
| 3 """Web tree checker. | |
| 4 | |
| 5 This utility is handy to check a subweb of the world-wide web for | |
| 6 errors. A subweb is specified by giving one or more ``root URLs''; a | |
| 7 page belongs to the subweb if one of the root URLs is an initial | |
| 8 prefix of it. | |
| 9 | |
| 10 File URL extension: | |
| 11 | |
| 12 In order to easy the checking of subwebs via the local file system, | |
| 13 the interpretation of ``file:'' URLs is extended to mimic the behavior | |
| 14 of your average HTTP daemon: if a directory pathname is given, the | |
| 15 file index.html in that directory is returned if it exists, otherwise | |
| 16 a directory listing is returned. Now, you can point webchecker to the | |
| 17 document tree in the local file system of your HTTP daemon, and have | |
| 18 most of it checked. In fact the default works this way if your local | |
| 19 web tree is located at /usr/local/etc/httpd/htdpcs (the default for | |
| 20 the NCSA HTTP daemon and probably others). | |
| 21 | |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
22 Report printed: |
| 5062 | 23 |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
24 When done, it reports pages with bad links within the subweb. When |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
25 interrupted, it reports for the pages that it has checked already. |
| 5062 | 26 |
| 27 In verbose mode, additional messages are printed during the | |
| 28 information gathering phase. By default, it prints a summary of its | |
| 29 work status every 50 URLs (adjustable with the -r option), and it | |
| 30 reports errors as they are encountered. Use the -q option to disable | |
| 31 this output. | |
| 32 | |
| 33 Checkpoint feature: | |
| 34 | |
| 35 Whether interrupted or not, it dumps its state (a Python pickle) to a | |
| 36 checkpoint file and the -R option allows it to restart from the | |
| 37 checkpoint (assuming that the pages on the subweb that were already | |
| 38 processed haven't changed). Even when it has run till completion, -R | |
| 39 can still be useful -- it will print the reports again, and -Rq prints | |
| 40 the errors only. In this case, the checkpoint file is not written | |
| 41 again. The checkpoint file can be set with the -d option. | |
| 42 | |
| 43 The checkpoint file is written as a Python pickle. Remember that | |
| 44 Python's pickle module is currently quite slow. Give it the time it | |
| 45 needs to load and save the checkpoint file. When interrupted while | |
| 46 writing the checkpoint file, the old checkpoint file is not | |
| 47 overwritten, but all work done in the current run is lost. | |
| 48 | |
| 49 Miscellaneous: | |
| 50 | |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
51 - You may find the (Tk-based) GUI version easier to use. See wcgui.py. |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
52 |
|
5064
7dded7851fca
Added robots.txt support, using Skip Montanaro's parser.
Guido van Rossum <guido@python.org>
parents:
5062
diff
changeset
|
53 - Webchecker honors the "robots.txt" convention. Thanks to Skip |
|
7dded7851fca
Added robots.txt support, using Skip Montanaro's parser.
Guido van Rossum <guido@python.org>
parents:
5062
diff
changeset
|
54 Montanaro for his robotparser.py module (included in this directory)! |
|
7dded7851fca
Added robots.txt support, using Skip Montanaro's parser.
Guido van Rossum <guido@python.org>
parents:
5062
diff
changeset
|
55 The agent name is hardwired to "webchecker". URLs that are disallowed |
|
7dded7851fca
Added robots.txt support, using Skip Montanaro's parser.
Guido van Rossum <guido@python.org>
parents:
5062
diff
changeset
|
56 by the robots.txt file are reported as external URLs. |
|
7dded7851fca
Added robots.txt support, using Skip Montanaro's parser.
Guido van Rossum <guido@python.org>
parents:
5062
diff
changeset
|
57 |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
58 - Because the SGML parser is a bit slow, very large SGML files are |
|
5064
7dded7851fca
Added robots.txt support, using Skip Montanaro's parser.
Guido van Rossum <guido@python.org>
parents:
5062
diff
changeset
|
59 skipped. The size limit can be set with the -m option. |
| 5062 | 60 |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
61 - When the server or protocol does not tell us a file's type, we guess |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
62 it based on the URL's suffix. The mimetypes.py module (also in this |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
63 directory) has a built-in table mapping most currently known suffixes, |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
64 and in addition attempts to read the mime.types configuration files in |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
65 the default locations of Netscape and the NCSA HTTP daemon. |
| 5062 | 66 |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
67 - We follows links indicated by <A>, <FRAME> and <IMG> tags. We also |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
68 honor the <BASE> tag. |
| 5062 | 69 |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
70 - Checking external links is now done by default; use -x to *disable* |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
71 this feature. External links are now checked during normal |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
72 processing. (XXX The status of a checked link could be categorized |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
73 better. Later...) |
| 5062 | 74 |
| 75 | |
| 76 Usage: webchecker.py [option] ... [rooturl] ... | |
| 77 | |
| 78 Options: | |
| 79 | |
| 80 -R -- restart from checkpoint file | |
| 81 -d file -- checkpoint filename (default %(DUMPFILE)s) | |
| 82 -m bytes -- skip HTML pages larger than this size (default %(MAXPAGE)d) | |
| 5076 | 83 -n -- reports only, no checking (use with -R) |
| 5062 | 84 -q -- quiet operation (also suppresses external links report) |
| 85 -r number -- number of links processed per round (default %(ROUNDSIZE)d) | |
| 86 -v -- verbose operation; repeating -v will increase verbosity | |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
87 -x -- don't check external links (these are often slow to check) |
| 5062 | 88 |
| 89 Arguments: | |
| 90 | |
| 91 rooturl -- URL to start checking | |
| 92 (default %(DEFROOT)s) | |
| 93 | |
| 94 """ | |
| 95 | |
| 5076 | 96 |
|
5423
bccb8d742964
Avoid the fancy handler for error 401 (request authentication).
Guido van Rossum <guido@python.org>
parents:
5086
diff
changeset
|
97 __version__ = "0.5" |
|
5066
77616b3001ef
Catch I/O errors when parsing robots.txt file.
Guido van Rossum <guido@python.org>
parents:
5064
diff
changeset
|
98 |
| 5062 | 99 |
| 100 import sys | |
| 101 import os | |
| 102 from types import * | |
| 103 import string | |
| 104 import StringIO | |
| 105 import getopt | |
| 106 import pickle | |
| 107 | |
| 108 import urllib | |
| 109 import urlparse | |
| 5076 | 110 import sgmllib |
| 5062 | 111 |
| 112 import mimetypes | |
|
5064
7dded7851fca
Added robots.txt support, using Skip Montanaro's parser.
Guido van Rossum <guido@python.org>
parents:
5062
diff
changeset
|
113 import robotparser |
| 5062 | 114 |
| 115 | |
| 116 # Tunable parameters | |
| 117 DEFROOT = "file:/usr/local/etc/httpd/htdocs/" # Default root URL | |
|
5080
9a8adaf58ce8
Spin off checking of external page in a subroutine.
Guido van Rossum <guido@python.org>
parents:
5076
diff
changeset
|
118 MAXPAGE = 150000 # Ignore files bigger than this |
| 5062 | 119 ROUNDSIZE = 50 # Number of links processed per round |
| 120 DUMPFILE = "@webchecker.pickle" # Pickled checkpoint | |
|
5064
7dded7851fca
Added robots.txt support, using Skip Montanaro's parser.
Guido van Rossum <guido@python.org>
parents:
5062
diff
changeset
|
121 AGENTNAME = "webchecker" # Agent name for robots.txt parser |
| 5062 | 122 |
| 123 | |
| 124 # Global variables | |
| 125 verbose = 1 | |
| 126 maxpage = MAXPAGE | |
| 127 roundsize = ROUNDSIZE | |
| 128 | |
| 129 | |
| 130 def main(): | |
| 131 global verbose, maxpage, roundsize | |
| 132 dumpfile = DUMPFILE | |
| 133 restart = 0 | |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
134 checkext = 1 |
| 5076 | 135 norun = 0 |
| 5062 | 136 |
| 137 try: | |
| 5076 | 138 opts, args = getopt.getopt(sys.argv[1:], 'Rd:m:nqr:vx') |
| 5062 | 139 except getopt.error, msg: |
| 140 sys.stdout = sys.stderr | |
| 141 print msg | |
|
5080
9a8adaf58ce8
Spin off checking of external page in a subroutine.
Guido van Rossum <guido@python.org>
parents:
5076
diff
changeset
|
142 print __doc__%globals() |
| 5062 | 143 sys.exit(2) |
| 144 for o, a in opts: | |
| 145 if o == '-R': | |
| 146 restart = 1 | |
| 147 if o == '-d': | |
| 148 dumpfile = a | |
| 149 if o == '-m': | |
| 150 maxpage = string.atoi(a) | |
| 5076 | 151 if o == '-n': |
| 152 norun = 1 | |
| 5062 | 153 if o == '-q': |
| 154 verbose = 0 | |
| 155 if o == '-r': | |
| 156 roundsize = string.atoi(a) | |
| 157 if o == '-v': | |
| 158 verbose = verbose + 1 | |
|
5067
4aaca42a2a16
Added -x option to check external links. Slooooow!
Guido van Rossum <guido@python.org>
parents:
5066
diff
changeset
|
159 if o == '-x': |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
160 checkext = not checkext |
| 5062 | 161 |
| 5076 | 162 if verbose > 0: |
|
5066
77616b3001ef
Catch I/O errors when parsing robots.txt file.
Guido van Rossum <guido@python.org>
parents:
5064
diff
changeset
|
163 print AGENTNAME, "version", __version__ |
|
77616b3001ef
Catch I/O errors when parsing robots.txt file.
Guido van Rossum <guido@python.org>
parents:
5064
diff
changeset
|
164 |
| 5062 | 165 if restart: |
| 166 if verbose > 0: | |
| 167 print "Loading checkpoint from %s ..." % dumpfile | |
| 168 f = open(dumpfile, "rb") | |
| 169 c = pickle.load(f) | |
| 170 f.close() | |
| 171 if verbose > 0: | |
| 172 print "Done." | |
| 173 print "Root:", string.join(c.roots, "\n ") | |
| 174 else: | |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
175 c = Checker(checkext) |
| 5062 | 176 if not args: |
| 177 args.append(DEFROOT) | |
| 178 | |
| 179 for arg in args: | |
| 180 c.addroot(arg) | |
| 181 | |
| 5076 | 182 if not norun: |
| 183 try: | |
| 184 c.run() | |
| 185 except KeyboardInterrupt: | |
| 186 if verbose > 0: | |
| 187 print "[run interrupted]" | |
| 188 | |
|
5067
4aaca42a2a16
Added -x option to check external links. Slooooow!
Guido van Rossum <guido@python.org>
parents:
5066
diff
changeset
|
189 try: |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
190 c.report() |
|
5067
4aaca42a2a16
Added -x option to check external links. Slooooow!
Guido van Rossum <guido@python.org>
parents:
5066
diff
changeset
|
191 except KeyboardInterrupt: |
|
4aaca42a2a16
Added -x option to check external links. Slooooow!
Guido van Rossum <guido@python.org>
parents:
5066
diff
changeset
|
192 if verbose > 0: |
|
4aaca42a2a16
Added -x option to check external links. Slooooow!
Guido van Rossum <guido@python.org>
parents:
5066
diff
changeset
|
193 print "[report interrupted]" |
| 5076 | 194 |
| 195 if not c.changed: | |
| 5062 | 196 if verbose > 0: |
| 197 print | |
| 198 print "No need to save checkpoint" | |
| 5076 | 199 elif not dumpfile: |
| 200 if verbose > 0: | |
| 201 print "No dumpfile, won't save checkpoint" | |
| 202 else: | |
| 5062 | 203 if verbose > 0: |
| 204 print | |
| 205 print "Saving checkpoint to %s ..." % dumpfile | |
| 206 newfile = dumpfile + ".new" | |
| 207 f = open(newfile, "wb") | |
| 208 pickle.dump(c, f) | |
| 209 f.close() | |
| 210 try: | |
| 211 os.unlink(dumpfile) | |
| 212 except os.error: | |
| 213 pass | |
| 214 os.rename(newfile, dumpfile) | |
| 215 if verbose > 0: | |
| 216 print "Done." | |
| 217 if dumpfile == DUMPFILE: | |
| 218 print "Use ``%s -R'' to restart." % sys.argv[0] | |
| 219 else: | |
| 220 print "Use ``%s -R -d %s'' to restart." % (sys.argv[0], | |
| 221 dumpfile) | |
| 222 | |
| 223 | |
| 224 class Checker: | |
| 225 | |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
226 def __init__(self, checkext=1): |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
227 self.reset() |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
228 self.checkext = checkext |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
229 |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
230 def reset(self): |
| 5062 | 231 self.roots = [] |
| 232 self.todo = {} | |
| 233 self.done = {} | |
| 234 self.bad = {} | |
| 5076 | 235 self.round = 0 |
| 236 # The following are not pickled: | |
| 237 self.robots = {} | |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
238 self.errors = {} |
| 5062 | 239 self.urlopener = MyURLopener() |
| 5076 | 240 self.changed = 0 |
|
5064
7dded7851fca
Added robots.txt support, using Skip Montanaro's parser.
Guido van Rossum <guido@python.org>
parents:
5062
diff
changeset
|
241 |
|
7dded7851fca
Added robots.txt support, using Skip Montanaro's parser.
Guido van Rossum <guido@python.org>
parents:
5062
diff
changeset
|
242 def __getstate__(self): |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
243 return (self.roots, self.todo, self.done, self.bad, self.round) |
|
5064
7dded7851fca
Added robots.txt support, using Skip Montanaro's parser.
Guido van Rossum <guido@python.org>
parents:
5062
diff
changeset
|
244 |
|
7dded7851fca
Added robots.txt support, using Skip Montanaro's parser.
Guido van Rossum <guido@python.org>
parents:
5062
diff
changeset
|
245 def __setstate__(self, state): |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
246 (self.roots, self.todo, self.done, self.bad, self.round) = state |
|
5064
7dded7851fca
Added robots.txt support, using Skip Montanaro's parser.
Guido van Rossum <guido@python.org>
parents:
5062
diff
changeset
|
247 for root in self.roots: |
|
7dded7851fca
Added robots.txt support, using Skip Montanaro's parser.
Guido van Rossum <guido@python.org>
parents:
5062
diff
changeset
|
248 self.addrobot(root) |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
249 for url in self.bad.keys(): |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
250 self.markerror(url) |
| 5062 | 251 |
| 252 def addroot(self, root): | |
| 253 if root not in self.roots: | |
| 254 self.roots.append(root) | |
|
5064
7dded7851fca
Added robots.txt support, using Skip Montanaro's parser.
Guido van Rossum <guido@python.org>
parents:
5062
diff
changeset
|
255 self.addrobot(root) |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
256 self.newlink(root, ("<root>", root)) |
|
5064
7dded7851fca
Added robots.txt support, using Skip Montanaro's parser.
Guido van Rossum <guido@python.org>
parents:
5062
diff
changeset
|
257 |
|
7dded7851fca
Added robots.txt support, using Skip Montanaro's parser.
Guido van Rossum <guido@python.org>
parents:
5062
diff
changeset
|
258 def addrobot(self, root): |
|
5066
77616b3001ef
Catch I/O errors when parsing robots.txt file.
Guido van Rossum <guido@python.org>
parents:
5064
diff
changeset
|
259 url = urlparse.urljoin(root, "/robots.txt") |
|
5064
7dded7851fca
Added robots.txt support, using Skip Montanaro's parser.
Guido van Rossum <guido@python.org>
parents:
5062
diff
changeset
|
260 self.robots[root] = rp = robotparser.RobotFileParser() |
|
5066
77616b3001ef
Catch I/O errors when parsing robots.txt file.
Guido van Rossum <guido@python.org>
parents:
5064
diff
changeset
|
261 if verbose > 2: |
|
77616b3001ef
Catch I/O errors when parsing robots.txt file.
Guido van Rossum <guido@python.org>
parents:
5064
diff
changeset
|
262 print "Parsing", url |
| 5076 | 263 rp.debug = verbose > 3 |
|
5064
7dded7851fca
Added robots.txt support, using Skip Montanaro's parser.
Guido van Rossum <guido@python.org>
parents:
5062
diff
changeset
|
264 rp.set_url(url) |
|
5066
77616b3001ef
Catch I/O errors when parsing robots.txt file.
Guido van Rossum <guido@python.org>
parents:
5064
diff
changeset
|
265 try: |
|
77616b3001ef
Catch I/O errors when parsing robots.txt file.
Guido van Rossum <guido@python.org>
parents:
5064
diff
changeset
|
266 rp.read() |
|
77616b3001ef
Catch I/O errors when parsing robots.txt file.
Guido van Rossum <guido@python.org>
parents:
5064
diff
changeset
|
267 except IOError, msg: |
|
77616b3001ef
Catch I/O errors when parsing robots.txt file.
Guido van Rossum <guido@python.org>
parents:
5064
diff
changeset
|
268 if verbose > 1: |
|
77616b3001ef
Catch I/O errors when parsing robots.txt file.
Guido van Rossum <guido@python.org>
parents:
5064
diff
changeset
|
269 print "I/O error parsing", url, ":", msg |
| 5062 | 270 |
| 271 def run(self): | |
| 272 while self.todo: | |
| 273 self.round = self.round + 1 | |
| 274 if verbose > 0: | |
| 275 print | |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
276 print "Round %d (%s)" % (self.round, self.status()) |
| 5076 | 277 print |
| 5062 | 278 urls = self.todo.keys()[:roundsize] |
| 279 for url in urls: | |
| 280 self.dopage(url) | |
| 5076 | 281 |
| 282 def status(self): | |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
283 return "%d total, %d to do, %d done, %d bad" % ( |
| 5076 | 284 len(self.todo)+len(self.done), |
| 285 len(self.todo), len(self.done), | |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
286 len(self.bad)) |
| 5062 | 287 |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
288 def report(self): |
| 5062 | 289 print |
| 290 if not self.todo: print "Final", | |
| 291 else: print "Interim", | |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
292 print "Report (%s)" % self.status() |
| 5062 | 293 self.report_errors() |
| 294 | |
| 295 def report_errors(self): | |
| 296 if not self.bad: | |
| 297 print | |
| 298 print "No errors" | |
| 299 return | |
| 300 print | |
| 301 print "Error Report:" | |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
302 sources = self.errors.keys() |
| 5062 | 303 sources.sort() |
| 304 for source in sources: | |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
305 triples = self.errors[source] |
| 5062 | 306 print |
| 307 if len(triples) > 1: | |
| 308 print len(triples), "Errors in", source | |
| 309 else: | |
| 310 print "Error in", source | |
| 311 for url, rawlink, msg in triples: | |
| 312 print " HREF", url, | |
| 313 if rawlink != url: print "(%s)" % rawlink, | |
| 314 print | |
| 315 print " msg", msg | |
| 316 | |
| 317 def dopage(self, url): | |
| 318 if verbose > 1: | |
| 319 if verbose > 2: | |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
320 show("Check ", url, " from", self.todo[url]) |
| 5062 | 321 else: |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
322 print "Check ", url |
| 5062 | 323 page = self.getpage(url) |
| 5076 | 324 if page: |
| 325 for info in page.getlinkinfos(): | |
| 326 link, rawlink = info | |
| 327 origin = url, rawlink | |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
328 self.newlink(link, origin) |
| 5076 | 329 self.markdone(url) |
| 330 | |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
331 def newlink(self, url, origin): |
| 5076 | 332 if self.done.has_key(url): |
| 333 self.newdonelink(url, origin) | |
| 334 else: | |
| 335 self.newtodolink(url, origin) | |
| 336 | |
| 337 def newdonelink(self, url, origin): | |
| 338 self.done[url].append(origin) | |
| 339 if verbose > 3: | |
| 340 print " Done link", url | |
| 341 | |
| 342 def newtodolink(self, url, origin): | |
| 343 if self.todo.has_key(url): | |
| 344 self.todo[url].append(origin) | |
| 345 if verbose > 3: | |
| 346 print " Seen todo link", url | |
| 347 else: | |
| 348 self.todo[url] = [origin] | |
| 349 if verbose > 3: | |
| 350 print " New todo link", url | |
| 351 | |
| 352 def markdone(self, url): | |
| 353 self.done[url] = self.todo[url] | |
| 354 del self.todo[url] | |
| 355 self.changed = 1 | |
| 5062 | 356 |
| 357 def inroots(self, url): | |
| 358 for root in self.roots: | |
| 359 if url[:len(root)] == root: | |
|
5064
7dded7851fca
Added robots.txt support, using Skip Montanaro's parser.
Guido van Rossum <guido@python.org>
parents:
5062
diff
changeset
|
360 return self.robots[root].can_fetch(AGENTNAME, url) |
| 5062 | 361 return 0 |
| 362 | |
| 363 def getpage(self, url): | |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
364 if url[:7] == 'mailto:' or url[:5] == 'news:': |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
365 if verbose > 1: print " Not checking mailto/news URL" |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
366 return None |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
367 isint = self.inroots(url) |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
368 if not isint and not self.checkext: |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
369 if verbose > 1: print " Not checking ext link" |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
370 return None |
| 5062 | 371 try: |
| 372 f = self.urlopener.open(url) | |
| 373 except IOError, msg: | |
|
5068
8ecf2d7c9fb1
Some refinements of the external-link checking code: insert the errors
Guido van Rossum <guido@python.org>
parents:
5067
diff
changeset
|
374 msg = sanitize(msg) |
| 5062 | 375 if verbose > 0: |
| 376 print "Error ", msg | |
| 377 if verbose > 0: | |
| 378 show(" HREF ", url, " from", self.todo[url]) | |
| 5076 | 379 self.setbad(url, msg) |
| 5062 | 380 return None |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
381 if not isint: |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
382 if verbose > 1: print " Not gathering links from ext URL" |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
383 safeclose(f) |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
384 return None |
| 5062 | 385 nurl = f.geturl() |
| 386 info = f.info() | |
| 387 if info.has_key('content-type'): | |
| 388 ctype = string.lower(info['content-type']) | |
| 5076 | 389 else: |
| 390 ctype = None | |
| 5062 | 391 if nurl != url: |
| 392 if verbose > 1: | |
|
5064
7dded7851fca
Added robots.txt support, using Skip Montanaro's parser.
Guido van Rossum <guido@python.org>
parents:
5062
diff
changeset
|
393 print " Redirected to", nurl |
| 5076 | 394 if not ctype: |
| 395 ctype, encoding = mimetypes.guess_type(nurl) | |
| 5062 | 396 if ctype != 'text/html': |
| 5076 | 397 safeclose(f) |
| 398 if verbose > 1: | |
| 399 print " Not HTML, mime type", ctype | |
| 5062 | 400 return None |
| 401 text = f.read() | |
| 402 f.close() | |
| 403 return Page(text, nurl) | |
| 404 | |
| 5076 | 405 def setgood(self, url): |
| 406 if self.bad.has_key(url): | |
| 407 del self.bad[url] | |
| 408 self.changed = 1 | |
| 409 if verbose > 0: | |
| 410 print "(Clear previously seen error)" | |
| 411 | |
| 412 def setbad(self, url, msg): | |
| 413 if self.bad.has_key(url) and self.bad[url] == msg: | |
| 414 if verbose > 0: | |
| 415 print "(Seen this error before)" | |
| 416 return | |
| 417 self.bad[url] = msg | |
| 418 self.changed = 1 | |
|
5086
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
419 self.markerror(url) |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
420 |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
421 def markerror(self, url): |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
422 try: |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
423 origins = self.todo[url] |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
424 except KeyError: |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
425 origins = self.done[url] |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
426 for source, rawlink in origins: |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
427 triple = url, rawlink, self.bad[url] |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
428 self.seterror(source, triple) |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
429 |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
430 def seterror(self, url, triple): |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
431 try: |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
432 self.errors[url].append(triple) |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
433 except KeyError: |
|
aa73da661cdf
Restructured Checker class to get rid of 'ext' table.
Guido van Rossum <guido@python.org>
parents:
5083
diff
changeset
|
434 self.errors[url] = [triple] |
| 5076 | 435 |
| 5062 | 436 |
| 437 class Page: | |
| 438 | |
| 439 def __init__(self, text, url): | |
| 440 self.text = text | |
| 441 self.url = url | |
| 442 | |
| 443 def getlinkinfos(self): | |
| 444 size = len(self.text) | |
| 445 if size > maxpage: | |
| 446 if verbose > 0: | |
| 447 print "Skip huge file", self.url | |
| 448 print " (%.0f Kbytes)" % (size*0.001) | |
| 449 return [] | |
| 450 if verbose > 2: | |
| 451 print " Parsing", self.url, "(%d bytes)" % size | |
| 5076 | 452 parser = MyHTMLParser() |
| 5062 | 453 parser.feed(self.text) |
| 454 parser.close() | |
| 455 rawlinks = parser.getlinks() | |
| 456 base = urlparse.urljoin(self.url, parser.getbase() or "") | |
| 457 infos = [] | |
| 458 for rawlink in rawlinks: | |
| 459 t = urlparse.urlparse(rawlink) | |
| 460 t = t[:-1] + ('',) | |
| 461 rawlink = urlparse.urlunparse(t) | |
| 462 link = urlparse.urljoin(base, rawlink) | |
| 463 infos.append((link, rawlink)) | |
| 464 return infos | |
| 465 | |
| 466 | |
| 467 class MyStringIO(StringIO.StringIO): | |
| 468 | |
| 469 def __init__(self, url, info): | |
| 470 self.__url = url | |
| 471 self.__info = info | |
| 472 StringIO.StringIO.__init__(self) | |
| 473 | |
| 474 def info(self): | |
| 475 return self.__info | |
| 476 | |
| 477 def geturl(self): | |
| 478 return self.__url | |
| 479 | |
| 480 | |
| 481 class MyURLopener(urllib.FancyURLopener): | |
| 482 | |
| 483 http_error_default = urllib.URLopener.http_error_default | |
| 484 | |
|
5069
fa31476a2786
Set proper User-agent header (Python-webchecker/<version>).
Guido van Rossum <guido@python.org>
parents:
5068
diff
changeset
|
485 def __init__(*args): |
|
fa31476a2786
Set proper User-agent header (Python-webchecker/<version>).
Guido van Rossum <guido@python.org>
parents:
5068
diff
changeset
|
486 self = args[0] |
|
fa31476a2786
Set proper User-agent header (Python-webchecker/<version>).
Guido van Rossum <guido@python.org>
parents:
5068
diff
changeset
|
487 apply(urllib.FancyURLopener.__init__, args) |
|
5423
bccb8d742964
Avoid the fancy handler for error 401 (request authentication).
Guido van Rossum <guido@python.org>
parents:
5086
diff
changeset
|
488 self.addheaders = [ |
|
bccb8d742964
Avoid the fancy handler for error 401 (request authentication).
Guido van Rossum <guido@python.org>
parents:
5086
diff
changeset
|
489 ('User-agent', 'Python-webchecker/%s' % __version__), |
|
bccb8d742964
Avoid the fancy handler for error 401 (request authentication).
Guido van Rossum <guido@python.org>
parents:
5086
diff
changeset
|
490 ] |
|
bccb8d742964
Avoid the fancy handler for error 401 (request authentication).
Guido van Rossum <guido@python.org>
parents:
5086
diff
changeset
|
491 |
|
bccb8d742964
Avoid the fancy handler for error 401 (request authentication).
Guido van Rossum <guido@python.org>
parents:
5086
diff
changeset
|
492 def http_error_401(self, url, fp, errcode, errmsg, headers): |
|
bccb8d742964
Avoid the fancy handler for error 401 (request authentication).
Guido van Rossum <guido@python.org>
parents:
5086
diff
changeset
|
493 return None |
|
5069
fa31476a2786
Set proper User-agent header (Python-webchecker/<version>).
Guido van Rossum <guido@python.org>
parents:
5068
diff
changeset
|
494 |
| 5062 | 495 def open_file(self, url): |
| 496 path = urllib.url2pathname(urllib.unquote(url)) | |
| 497 if path[-1] != os.sep: | |
| 498 url = url + '/' | |
| 499 if os.path.isdir(path): | |
| 500 indexpath = os.path.join(path, "index.html") | |
| 501 if os.path.exists(indexpath): | |
| 502 return self.open_file(url + "index.html") | |
| 503 try: | |
| 504 names = os.listdir(path) | |
| 505 except os.error, msg: | |
| 506 raise IOError, msg, sys.exc_traceback | |
| 507 names.sort() | |
| 508 s = MyStringIO("file:"+url, {'content-type': 'text/html'}) | |
| 509 s.write('<BASE HREF="file:%s">\n' % | |
| 510 urllib.quote(os.path.join(path, ""))) | |
| 511 for name in names: | |
| 512 q = urllib.quote(name) | |
| 513 s.write('<A HREF="%s">%s</A>\n' % (q, q)) | |
| 514 s.seek(0) | |
| 515 return s | |
| 516 return urllib.FancyURLopener.open_file(self, path) | |
| 517 | |
| 518 | |
| 5076 | 519 class MyHTMLParser(sgmllib.SGMLParser): |
| 5062 | 520 |
| 5076 | 521 def __init__(self): |
| 5062 | 522 self.base = None |
| 5076 | 523 self.links = {} |
| 524 sgmllib.SGMLParser.__init__ (self) | |
| 5062 | 525 |
| 526 def start_a(self, attributes): | |
|
5083
7ab1aa08d348
Process <img> and <frame> tags. Don't bother skipping second href.
Guido van Rossum <guido@python.org>
parents:
5080
diff
changeset
|
527 self.link_attr(attributes, 'href') |
|
7ab1aa08d348
Process <img> and <frame> tags. Don't bother skipping second href.
Guido van Rossum <guido@python.org>
parents:
5080
diff
changeset
|
528 |
|
7ab1aa08d348
Process <img> and <frame> tags. Don't bother skipping second href.
Guido van Rossum <guido@python.org>
parents:
5080
diff
changeset
|
529 def end_a(self): pass |
|
7ab1aa08d348
Process <img> and <frame> tags. Don't bother skipping second href.
Guido van Rossum <guido@python.org>
parents:
5080
diff
changeset
|
530 |
|
7ab1aa08d348
Process <img> and <frame> tags. Don't bother skipping second href.
Guido van Rossum <guido@python.org>
parents:
5080
diff
changeset
|
531 def do_img(self, attributes): |
|
7ab1aa08d348
Process <img> and <frame> tags. Don't bother skipping second href.
Guido van Rossum <guido@python.org>
parents:
5080
diff
changeset
|
532 self.link_attr(attributes, 'src', 'lowsrc') |
|
7ab1aa08d348
Process <img> and <frame> tags. Don't bother skipping second href.
Guido van Rossum <guido@python.org>
parents:
5080
diff
changeset
|
533 |
|
7ab1aa08d348
Process <img> and <frame> tags. Don't bother skipping second href.
Guido van Rossum <guido@python.org>
parents:
5080
diff
changeset
|
534 def do_frame(self, attributes): |
|
7ab1aa08d348
Process <img> and <frame> tags. Don't bother skipping second href.
Guido van Rossum <guido@python.org>
parents:
5080
diff
changeset
|
535 self.link_attr(attributes, 'src') |
|
7ab1aa08d348
Process <img> and <frame> tags. Don't bother skipping second href.
Guido van Rossum <guido@python.org>
parents:
5080
diff
changeset
|
536 |
|
7ab1aa08d348
Process <img> and <frame> tags. Don't bother skipping second href.
Guido van Rossum <guido@python.org>
parents:
5080
diff
changeset
|
537 def link_attr(self, attributes, *args): |
| 5062 | 538 for name, value in attributes: |
|
5083
7ab1aa08d348
Process <img> and <frame> tags. Don't bother skipping second href.
Guido van Rossum <guido@python.org>
parents:
5080
diff
changeset
|
539 if name in args: |
| 5076 | 540 if value: value = string.strip(value) |
| 541 if value: self.links[value] = None | |
| 5062 | 542 |
| 543 def do_base(self, attributes): | |
| 544 for name, value in attributes: | |
| 5076 | 545 if name == 'href': |
| 546 if value: value = string.strip(value) | |
| 547 if value: | |
| 548 if verbose > 1: | |
| 549 print " Base", value | |
| 550 self.base = value | |
| 5062 | 551 |
| 552 def getlinks(self): | |
| 5076 | 553 return self.links.keys() |
| 5062 | 554 |
| 555 def getbase(self): | |
| 556 return self.base | |
| 557 | |
| 558 | |
| 559 def show(p1, link, p2, origins): | |
| 560 print p1, link | |
| 561 i = 0 | |
| 562 for source, rawlink in origins: | |
| 563 i = i+1 | |
| 564 if i == 2: | |
| 565 p2 = ' '*len(p2) | |
| 566 print p2, source, | |
| 567 if rawlink != link: print "(%s)" % rawlink, | |
| 568 print | |
| 569 | |
| 570 | |
|
5068
8ecf2d7c9fb1
Some refinements of the external-link checking code: insert the errors
Guido van Rossum <guido@python.org>
parents:
5067
diff
changeset
|
571 def sanitize(msg): |
|
8ecf2d7c9fb1
Some refinements of the external-link checking code: insert the errors
Guido van Rossum <guido@python.org>
parents:
5067
diff
changeset
|
572 if (type(msg) == TupleType and |
|
8ecf2d7c9fb1
Some refinements of the external-link checking code: insert the errors
Guido van Rossum <guido@python.org>
parents:
5067
diff
changeset
|
573 len(msg) >= 4 and |
|
8ecf2d7c9fb1
Some refinements of the external-link checking code: insert the errors
Guido van Rossum <guido@python.org>
parents:
5067
diff
changeset
|
574 msg[0] == 'http error' and |
|
8ecf2d7c9fb1
Some refinements of the external-link checking code: insert the errors
Guido van Rossum <guido@python.org>
parents:
5067
diff
changeset
|
575 type(msg[3]) == InstanceType): |
|
8ecf2d7c9fb1
Some refinements of the external-link checking code: insert the errors
Guido van Rossum <guido@python.org>
parents:
5067
diff
changeset
|
576 # Remove the Message instance -- it may contain |
|
8ecf2d7c9fb1
Some refinements of the external-link checking code: insert the errors
Guido van Rossum <guido@python.org>
parents:
5067
diff
changeset
|
577 # a file object which prevents pickling. |
|
8ecf2d7c9fb1
Some refinements of the external-link checking code: insert the errors
Guido van Rossum <guido@python.org>
parents:
5067
diff
changeset
|
578 msg = msg[:3] + msg[4:] |
|
8ecf2d7c9fb1
Some refinements of the external-link checking code: insert the errors
Guido van Rossum <guido@python.org>
parents:
5067
diff
changeset
|
579 return msg |
|
8ecf2d7c9fb1
Some refinements of the external-link checking code: insert the errors
Guido van Rossum <guido@python.org>
parents:
5067
diff
changeset
|
580 |
|
8ecf2d7c9fb1
Some refinements of the external-link checking code: insert the errors
Guido van Rossum <guido@python.org>
parents:
5067
diff
changeset
|
581 |
| 5076 | 582 def safeclose(f): |
| 583 url = f.geturl() | |
| 584 if url[:4] == 'ftp:' or url[:7] == 'file://': | |
| 585 # Apparently ftp connections don't like to be closed | |
| 586 # prematurely... | |
| 587 text = f.read() | |
| 588 f.close() | |
| 589 | |
| 590 | |
| 5062 | 591 if __name__ == '__main__': |
| 592 main() |