cpython-lru_cache-weakref: Tools/webchecker/webchecker.py annotate

annotate Tools/webchecker/webchecker.py @ 54294:dba767bc3155 legacy-trunk

remove mention of old ctypes version

author	Benjamin Peterson <benjamin@python.org>
date	Sat, 23 May 2009 20:59:09 +0000
parents	b81642e52287
children

rev	line source
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	1 #! /usr/bin/env python
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	2
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	3 # Original code by Guido van Rossum; extensive changes by Sam Bayer,
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	4 # including code to check URL fragments.
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	5
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	6 """Web tree checker.
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	7
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	8 This utility is handy to check a subweb of the world-wide web for
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	9 errors. A subweb is specified by giving one or more ``root URLs''; a
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	10 page belongs to the subweb if one of the root URLs is an initial
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	11 prefix of it.
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	12
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	13 File URL extension:
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	14
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	15 In order to easy the checking of subwebs via the local file system,
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	16 the interpretation of ``file:'' URLs is extended to mimic the behavior
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	17 of your average HTTP daemon: if a directory pathname is given, the
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	18 file index.html in that directory is returned if it exists, otherwise
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	19 a directory listing is returned. Now, you can point webchecker to the
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	20 document tree in the local file system of your HTTP daemon, and have
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	21 most of it checked. In fact the default works this way if your local
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	22 web tree is located at /usr/local/etc/httpd/htdpcs (the default for
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	23 the NCSA HTTP daemon and probably others).
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	24
5086 aa73da661cdf Restructured Checker class to get rid of 'ext' table. Guido van Rossum <guido@python.org> parents: 5083 diff changeset	25 Report printed:
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	26
5086 aa73da661cdf Restructured Checker class to get rid of 'ext' table. Guido van Rossum <guido@python.org> parents: 5083 diff changeset	27 When done, it reports pages with bad links within the subweb. When
aa73da661cdf Restructured Checker class to get rid of 'ext' table. Guido van Rossum <guido@python.org> parents: 5083 diff changeset	28 interrupted, it reports for the pages that it has checked already.
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	29
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	30 In verbose mode, additional messages are printed during the
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	31 information gathering phase. By default, it prints a summary of its
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	32 work status every 50 URLs (adjustable with the -r option), and it
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	33 reports errors as they are encountered. Use the -q option to disable
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	34 this output.
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	35
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	36 Checkpoint feature:
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	37
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	38 Whether interrupted or not, it dumps its state (a Python pickle) to a
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	39 checkpoint file and the -R option allows it to restart from the
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	40 checkpoint (assuming that the pages on the subweb that were already
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	41 processed haven't changed). Even when it has run till completion, -R
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	42 can still be useful -- it will print the reports again, and -Rq prints
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	43 the errors only. In this case, the checkpoint file is not written
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	44 again. The checkpoint file can be set with the -d option.
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	45
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	46 The checkpoint file is written as a Python pickle. Remember that
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	47 Python's pickle module is currently quite slow. Give it the time it
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	48 needs to load and save the checkpoint file. When interrupted while
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	49 writing the checkpoint file, the old checkpoint file is not
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	50 overwritten, but all work done in the current run is lost.
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	51
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	52 Miscellaneous:
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	53
5086 aa73da661cdf Restructured Checker class to get rid of 'ext' table. Guido van Rossum <guido@python.org> parents: 5083 diff changeset	54 - You may find the (Tk-based) GUI version easier to use. See wcgui.py.
aa73da661cdf Restructured Checker class to get rid of 'ext' table. Guido van Rossum <guido@python.org> parents: 5083 diff changeset	55
5064 7dded7851fca Added robots.txt support, using Skip Montanaro's parser. Guido van Rossum <guido@python.org> parents: 5062 diff changeset	56 - Webchecker honors the "robots.txt" convention. Thanks to Skip
7dded7851fca Added robots.txt support, using Skip Montanaro's parser. Guido van Rossum <guido@python.org> parents: 5062 diff changeset	57 Montanaro for his robotparser.py module (included in this directory)!
7dded7851fca Added robots.txt support, using Skip Montanaro's parser. Guido van Rossum <guido@python.org> parents: 5062 diff changeset	58 The agent name is hardwired to "webchecker". URLs that are disallowed
7dded7851fca Added robots.txt support, using Skip Montanaro's parser. Guido van Rossum <guido@python.org> parents: 5062 diff changeset	59 by the robots.txt file are reported as external URLs.
7dded7851fca Added robots.txt support, using Skip Montanaro's parser. Guido van Rossum <guido@python.org> parents: 5062 diff changeset	60
5086 aa73da661cdf Restructured Checker class to get rid of 'ext' table. Guido van Rossum <guido@python.org> parents: 5083 diff changeset	61 - Because the SGML parser is a bit slow, very large SGML files are
5064 7dded7851fca Added robots.txt support, using Skip Montanaro's parser. Guido van Rossum <guido@python.org> parents: 5062 diff changeset	62 skipped. The size limit can be set with the -m option.
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	63
5086 aa73da661cdf Restructured Checker class to get rid of 'ext' table. Guido van Rossum <guido@python.org> parents: 5083 diff changeset	64 - When the server or protocol does not tell us a file's type, we guess
aa73da661cdf Restructured Checker class to get rid of 'ext' table. Guido van Rossum <guido@python.org> parents: 5083 diff changeset	65 it based on the URL's suffix. The mimetypes.py module (also in this
aa73da661cdf Restructured Checker class to get rid of 'ext' table. Guido van Rossum <guido@python.org> parents: 5083 diff changeset	66 directory) has a built-in table mapping most currently known suffixes,
aa73da661cdf Restructured Checker class to get rid of 'ext' table. Guido van Rossum <guido@python.org> parents: 5083 diff changeset	67 and in addition attempts to read the mime.types configuration files in
aa73da661cdf Restructured Checker class to get rid of 'ext' table. Guido van Rossum <guido@python.org> parents: 5083 diff changeset	68 the default locations of Netscape and the NCSA HTTP daemon.
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	69
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	70 - We follow links indicated by <A>, <FRAME> and <IMG> tags. We also
5086 aa73da661cdf Restructured Checker class to get rid of 'ext' table. Guido van Rossum <guido@python.org> parents: 5083 diff changeset	71 honor the <BASE> tag.
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	72
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	73 - We now check internal NAME anchor links, as well as toplevel links.
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	74
5086 aa73da661cdf Restructured Checker class to get rid of 'ext' table. Guido van Rossum <guido@python.org> parents: 5083 diff changeset	75 - Checking external links is now done by default; use -x to disable
aa73da661cdf Restructured Checker class to get rid of 'ext' table. Guido van Rossum <guido@python.org> parents: 5083 diff changeset	76 this feature. External links are now checked during normal
aa73da661cdf Restructured Checker class to get rid of 'ext' table. Guido van Rossum <guido@python.org> parents: 5083 diff changeset	77 processing. (XXX The status of a checked link could be categorized
aa73da661cdf Restructured Checker class to get rid of 'ext' table. Guido van Rossum <guido@python.org> parents: 5083 diff changeset	78 better. Later...)
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	79
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	80 - If external links are not checked, you can use the -t flag to
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	81 provide specific overrides to -x.
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	82
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	83 Usage: webchecker.py [option] ... [rooturl] ...
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	84
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	85 Options:
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	86
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	87 -R -- restart from checkpoint file
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	88 -d file -- checkpoint filename (default %(DUMPFILE)s)
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	89 -m bytes -- skip HTML pages larger than this size (default %(MAXPAGE)d)
5076 4edab474974c Many misc changes. Guido van Rossum <guido@python.org> parents: 5069 diff changeset	90 -n -- reports only, no checking (use with -R)
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	91 -q -- quiet operation (also suppresses external links report)
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	92 -r number -- number of links processed per round (default %(ROUNDSIZE)d)
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	93 -t root -- specify root dir which should be treated as internal (can repeat)
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	94 -v -- verbose operation; repeating -v will increase verbosity
5086 aa73da661cdf Restructured Checker class to get rid of 'ext' table. Guido van Rossum <guido@python.org> parents: 5083 diff changeset	95 -x -- don't check external links (these are often slow to check)
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	96 -a -- don't check name anchors
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	97
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	98 Arguments:
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	99
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	100 rooturl -- URL to start checking
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	101 (default %(DEFROOT)s)
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	102
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	103 """
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	104
5076 4edab474974c Many misc changes. Guido van Rossum <guido@python.org> parents: 5069 diff changeset	105
7447 826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	106 __version__ = "$Revision$"
5066 77616b3001ef Catch I/O errors when parsing robots.txt file. Guido van Rossum <guido@python.org> parents: 5064 diff changeset	107
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	108
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	109 import sys
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	110 import os
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	111 from types import *
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	112 import StringIO
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	113 import getopt
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	114 import pickle
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	115
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	116 import urllib
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	117 import urlparse
5076 4edab474974c Many misc changes. Guido van Rossum <guido@python.org> parents: 5069 diff changeset	118 import sgmllib
23704 e4add7bb5f14 Apply diff.txt from SF patch http://www.python.org/sf/561478 Walter Dörwald <walter@livinglogic.de> parents: 22219 diff changeset	119 import cgi
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	120
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	121 import mimetypes
5064 7dded7851fca Added robots.txt support, using Skip Montanaro's parser. Guido van Rossum <guido@python.org> parents: 5062 diff changeset	122 import robotparser
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	123
7447 826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	124 # Extract real version number if necessary
826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	125 if __version__[0] == '$':
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 23704 diff changeset	126 _v = __version__.split()
7447 826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	127 if len(_v) == 3:
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	128 __version__ = _v[1]
7447 826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	129
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	130
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	131 # Tunable parameters
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	132 DEFROOT = "file:/usr/local/etc/httpd/htdocs/" # Default root URL
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	133 CHECKEXT = 1 # Check external references (1 deep)
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	134 VERBOSE = 1 # Verbosity level (0-3)
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	135 MAXPAGE = 150000 # Ignore files bigger than this
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	136 ROUNDSIZE = 50 # Number of links processed per round
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	137 DUMPFILE = "@webchecker.pickle" # Pickled checkpoint
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	138 AGENTNAME = "webchecker" # Agent name for robots.txt parser
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	139 NONAMES = 0 # Force name anchor checking
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	140
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	141
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	142 # Global variables
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	143
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	144
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	145 def main():
7447 826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	146 checkext = CHECKEXT
826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	147 verbose = VERBOSE
826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	148 maxpage = MAXPAGE
826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	149 roundsize = ROUNDSIZE
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	150 dumpfile = DUMPFILE
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	151 restart = 0
5076 4edab474974c Many misc changes. Guido van Rossum <guido@python.org> parents: 5069 diff changeset	152 norun = 0
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	153
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	154 try:
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	155 opts, args = getopt.getopt(sys.argv[1:], 'Rd:m:nqr:t:vxa')
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	156 except getopt.error, msg:
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	157 sys.stdout = sys.stderr
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	158 print msg
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	159 print __doc__%globals()
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	160 sys.exit(2)
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	161
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	162 # The extra_roots variable collects extra roots.
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	163 extra_roots = []
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	164 nonames = NONAMES
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	165
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	166 for o, a in opts:
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	167 if o == '-R':
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	168 restart = 1
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	169 if o == '-d':
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	170 dumpfile = a
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	171 if o == '-m':
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 23704 diff changeset	172 maxpage = int(a)
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	173 if o == '-n':
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	174 norun = 1
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	175 if o == '-q':
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	176 verbose = 0
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	177 if o == '-r':
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 23704 diff changeset	178 roundsize = int(a)
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	179 if o == '-t':
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	180 extra_roots.append(a)
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	181 if o == '-a':
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	182 nonames = not nonames
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	183 if o == '-v':
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	184 verbose = verbose + 1
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	185 if o == '-x':
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	186 checkext = not checkext
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	187
5076 4edab474974c Many misc changes. Guido van Rossum <guido@python.org> parents: 5069 diff changeset	188 if verbose > 0:
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	189 print AGENTNAME, "version", __version__
5066 77616b3001ef Catch I/O errors when parsing robots.txt file. Guido van Rossum <guido@python.org> parents: 5064 diff changeset	190
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	191 if restart:
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	192 c = load_pickle(dumpfile=dumpfile, verbose=verbose)
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	193 else:
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	194 c = Checker()
7447 826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	195
826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	196 c.setflags(checkext=checkext, verbose=verbose,
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	197 maxpage=maxpage, roundsize=roundsize,
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	198 nonames=nonames
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	199 )
7447 826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	200
826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	201 if not restart and not args:
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	202 args.append(DEFROOT)
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	203
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	204 for arg in args:
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	205 c.addroot(arg)
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	206
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	207 # The -t flag is only needed if external links are not to be
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	208 # checked. So -t values are ignored unless -x was specified.
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	209 if not checkext:
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	210 for root in extra_roots:
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	211 # Make sure it's terminated by a slash,
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	212 # so that addroot doesn't discard the last
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	213 # directory component.
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	214 if root[-1] != "/":
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	215 root = root + "/"
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	216 c.addroot(root, add_to_do = 0)
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	217
8163 a3134c6970eb Use a try-except so that the pickle file is written even when we die Guido van Rossum <guido@python.org> parents: 7887 diff changeset	218 try:
a3134c6970eb Use a try-except so that the pickle file is written even when we die Guido van Rossum <guido@python.org> parents: 7887 diff changeset	219
a3134c6970eb Use a try-except so that the pickle file is written even when we die Guido van Rossum <guido@python.org> parents: 7887 diff changeset	220 if not norun:
a3134c6970eb Use a try-except so that the pickle file is written even when we die Guido van Rossum <guido@python.org> parents: 7887 diff changeset	221 try:
a3134c6970eb Use a try-except so that the pickle file is written even when we die Guido van Rossum <guido@python.org> parents: 7887 diff changeset	222 c.run()
a3134c6970eb Use a try-except so that the pickle file is written even when we die Guido van Rossum <guido@python.org> parents: 7887 diff changeset	223 except KeyboardInterrupt:
a3134c6970eb Use a try-except so that the pickle file is written even when we die Guido van Rossum <guido@python.org> parents: 7887 diff changeset	224 if verbose > 0:
a3134c6970eb Use a try-except so that the pickle file is written even when we die Guido van Rossum <guido@python.org> parents: 7887 diff changeset	225 print "[run interrupted]"
a3134c6970eb Use a try-except so that the pickle file is written even when we die Guido van Rossum <guido@python.org> parents: 7887 diff changeset	226
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	227 try:
8163 a3134c6970eb Use a try-except so that the pickle file is written even when we die Guido van Rossum <guido@python.org> parents: 7887 diff changeset	228 c.report()
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	229 except KeyboardInterrupt:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	230 if verbose > 0:
8163 a3134c6970eb Use a try-except so that the pickle file is written even when we die Guido van Rossum <guido@python.org> parents: 7887 diff changeset	231 print "[report interrupted]"
5076 4edab474974c Many misc changes. Guido van Rossum <guido@python.org> parents: 5069 diff changeset	232
8163 a3134c6970eb Use a try-except so that the pickle file is written even when we die Guido van Rossum <guido@python.org> parents: 7887 diff changeset	233 finally:
a3134c6970eb Use a try-except so that the pickle file is written even when we die Guido van Rossum <guido@python.org> parents: 7887 diff changeset	234 if c.save_pickle(dumpfile):
a3134c6970eb Use a try-except so that the pickle file is written even when we die Guido van Rossum <guido@python.org> parents: 7887 diff changeset	235 if dumpfile == DUMPFILE:
a3134c6970eb Use a try-except so that the pickle file is written even when we die Guido van Rossum <guido@python.org> parents: 7887 diff changeset	236 print "Use ``%s -R'' to restart." % sys.argv[0]
a3134c6970eb Use a try-except so that the pickle file is written even when we die Guido van Rossum <guido@python.org> parents: 7887 diff changeset	237 else:
a3134c6970eb Use a try-except so that the pickle file is written even when we die Guido van Rossum <guido@python.org> parents: 7887 diff changeset	238 print "Use ``%s -R -d %s'' to restart." % (sys.argv[0],
a3134c6970eb Use a try-except so that the pickle file is written even when we die Guido van Rossum <guido@python.org> parents: 7887 diff changeset	239 dumpfile)
7447 826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	240
826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	241
826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	242 def load_pickle(dumpfile=DUMPFILE, verbose=VERBOSE):
826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	243 if verbose > 0:
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	244 print "Loading checkpoint from %s ..." % dumpfile
7447 826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	245 f = open(dumpfile, "rb")
826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	246 c = pickle.load(f)
826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	247 f.close()
826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	248 if verbose > 0:
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	249 print "Done."
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 23704 diff changeset	250 print "Root:", "\n ".join(c.roots)
7447 826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	251 return c
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	252
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	253
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	254 class Checker:
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	255
7447 826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	256 checkext = CHECKEXT
826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	257 verbose = VERBOSE
826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	258 maxpage = MAXPAGE
826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	259 roundsize = ROUNDSIZE
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	260 nonames = NONAMES
7447 826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	261
826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	262 validflags = tuple(dir())
826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	263
826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	264 def __init__(self):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	265 self.reset()
7447 826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	266
826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	267 def setflags(self, **kw):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	268 for key in kw.keys():
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	269 if key not in self.validflags:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	270 raise NameError, "invalid keyword argument: %s" % str(key)
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	271 for key, value in kw.items():
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	272 setattr(self, key, value)
5086 aa73da661cdf Restructured Checker class to get rid of 'ext' table. Guido van Rossum <guido@python.org> parents: 5083 diff changeset	273
aa73da661cdf Restructured Checker class to get rid of 'ext' table. Guido van Rossum <guido@python.org> parents: 5083 diff changeset	274 def reset(self):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	275 self.roots = []
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	276 self.todo = {}
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	277 self.done = {}
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	278 self.bad = {}
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	279
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	280 # Add a name table, so that the name URLs can be checked. Also
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	281 # serves as an implicit cache for which URLs are done.
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	282 self.name_table = {}
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	283
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	284 self.round = 0
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	285 # The following are not pickled:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	286 self.robots = {}
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	287 self.errors = {}
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	288 self.urlopener = MyURLopener()
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	289 self.changed = 0
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	290
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	291 def note(self, level, format, *args):
a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	292 if self.verbose > level:
a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	293 if args:
a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	294 format = format%args
a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	295 self.message(format)
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	296
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	297 def message(self, format, *args):
a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	298 if args:
a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	299 format = format%args
32736 c3a096bfbd6b Whitespace normalization, via reindent.py. Tim Peters <tim.peters@gmail.com> parents: 31743 diff changeset	300 print format
5064 7dded7851fca Added robots.txt support, using Skip Montanaro's parser. Guido van Rossum <guido@python.org> parents: 5062 diff changeset	301
7dded7851fca Added robots.txt support, using Skip Montanaro's parser. Guido van Rossum <guido@python.org> parents: 5062 diff changeset	302 def __getstate__(self):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	303 return (self.roots, self.todo, self.done, self.bad, self.round)
5064 7dded7851fca Added robots.txt support, using Skip Montanaro's parser. Guido van Rossum <guido@python.org> parents: 5062 diff changeset	304
7dded7851fca Added robots.txt support, using Skip Montanaro's parser. Guido van Rossum <guido@python.org> parents: 5062 diff changeset	305 def __setstate__(self, state):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	306 self.reset()
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	307 (self.roots, self.todo, self.done, self.bad, self.round) = state
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	308 for root in self.roots:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	309 self.addrobot(root)
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	310 for url in self.bad.keys():
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	311 self.markerror(url)
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	312
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	313 def addroot(self, root, add_to_do = 1):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	314 if root not in self.roots:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	315 troot = root
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	316 scheme, netloc, path, params, query, fragment = \
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	317 urlparse.urlparse(root)
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 23704 diff changeset	318 i = path.rfind("/") + 1
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	319 if 0 < i < len(path):
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	320 path = path[:i]
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	321 troot = urlparse.urlunparse((scheme, netloc, path,
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	322 params, query, fragment))
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	323 self.roots.append(troot)
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	324 self.addrobot(root)
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	325 if add_to_do:
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	326 self.newlink((root, ""), ("<root>", root))
5064 7dded7851fca Added robots.txt support, using Skip Montanaro's parser. Guido van Rossum <guido@python.org> parents: 5062 diff changeset	327
7dded7851fca Added robots.txt support, using Skip Montanaro's parser. Guido van Rossum <guido@python.org> parents: 5062 diff changeset	328 def addrobot(self, root):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	329 root = urlparse.urljoin(root, "/")
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	330 if self.robots.has_key(root): return
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	331 url = urlparse.urljoin(root, "/robots.txt")
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	332 self.robots[root] = rp = robotparser.RobotFileParser()
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	333 self.note(2, "Parsing %s", url)
a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	334 rp.debug = self.verbose > 3
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	335 rp.set_url(url)
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	336 try:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	337 rp.read()
21361 e8b098fe3df8 Fix SF bug #482171: webchecker dies on file: URLs w/o robots.txt Guido van Rossum <guido@python.org> parents: 17318 diff changeset	338 except (OSError, IOError), msg:
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	339 self.note(1, "I/O error parsing %s: %s", url, msg)
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	340
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	341 def run(self):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	342 while self.todo:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	343 self.round = self.round + 1
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	344 self.note(0, "\nRound %d (%s)\n", self.round, self.status())
8501 ba312e790b2a sort the urls in the todo list Guido van Rossum <guido@python.org> parents: 8163 diff changeset	345 urls = self.todo.keys()
ba312e790b2a sort the urls in the todo list Guido van Rossum <guido@python.org> parents: 8163 diff changeset	346 urls.sort()
ba312e790b2a sort the urls in the todo list Guido van Rossum <guido@python.org> parents: 8163 diff changeset	347 del urls[self.roundsize:]
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	348 for url in urls:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	349 self.dopage(url)
5076 4edab474974c Many misc changes. Guido van Rossum <guido@python.org> parents: 5069 diff changeset	350
4edab474974c Many misc changes. Guido van Rossum <guido@python.org> parents: 5069 diff changeset	351 def status(self):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	352 return "%d total, %d to do, %d done, %d bad" % (
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	353 len(self.todo)+len(self.done),
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	354 len(self.todo), len(self.done),
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	355 len(self.bad))
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	356
5086 aa73da661cdf Restructured Checker class to get rid of 'ext' table. Guido van Rossum <guido@python.org> parents: 5083 diff changeset	357 def report(self):
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	358 self.message("")
a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	359 if not self.todo: s = "Final"
a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	360 else: s = "Interim"
a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	361 self.message("%s Report (%s)", s, self.status())
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	362 self.report_errors()
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	363
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	364 def report_errors(self):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	365 if not self.bad:
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	366 self.message("\nNo errors")
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	367 return
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	368 self.message("\nError Report:")
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	369 sources = self.errors.keys()
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	370 sources.sort()
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	371 for source in sources:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	372 triples = self.errors[source]
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	373 self.message("")
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	374 if len(triples) > 1:
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	375 self.message("%d Errors in %s", len(triples), source)
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	376 else:
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	377 self.message("Error in %s", source)
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	378 # Call self.format_url() instead of referring
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	379 # to the URL directly, since the URLs in these
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	380 # triples is now a (URL, fragment) pair. The value
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	381 # of the "source" variable comes from the list of
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	382 # origins, and is a URL, not a pair.
32736 c3a096bfbd6b Whitespace normalization, via reindent.py. Tim Peters <tim.peters@gmail.com> parents: 31743 diff changeset	383 for url, rawlink, msg in triples:
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	384 if rawlink != self.format_url(url): s = " (%s)" % rawlink
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	385 else: s = ""
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	386 self.message(" HREF %s%s\n msg %s",
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	387 self.format_url(url), s, msg)
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	388
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	389 def dopage(self, url_pair):
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	390
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	391 # All printing of URLs uses format_url(); argument changed to
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	392 # url_pair for clarity.
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	393 if self.verbose > 1:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	394 if self.verbose > 2:
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	395 self.show("Check ", self.format_url(url_pair),
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	396 " from", self.todo[url_pair])
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	397 else:
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	398 self.message("Check %s", self.format_url(url_pair))
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	399 url, local_fragment = url_pair
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	400 if local_fragment and self.nonames:
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	401 self.markdone(url_pair)
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	402 return
28085 f42fbc7c7d8e When bad HTML is encountered, ignore the page rather than failing with Mark Hammond <mhammond@skippinet.com.au> parents: 26162 diff changeset	403 try:
f42fbc7c7d8e When bad HTML is encountered, ignore the page rather than failing with Mark Hammond <mhammond@skippinet.com.au> parents: 26162 diff changeset	404 page = self.getpage(url_pair)
f42fbc7c7d8e When bad HTML is encountered, ignore the page rather than failing with Mark Hammond <mhammond@skippinet.com.au> parents: 26162 diff changeset	405 except sgmllib.SGMLParseError, msg:
f42fbc7c7d8e When bad HTML is encountered, ignore the page rather than failing with Mark Hammond <mhammond@skippinet.com.au> parents: 26162 diff changeset	406 msg = self.sanitize(msg)
f42fbc7c7d8e When bad HTML is encountered, ignore the page rather than failing with Mark Hammond <mhammond@skippinet.com.au> parents: 26162 diff changeset	407 self.note(0, "Error parsing %s: %s",
f42fbc7c7d8e When bad HTML is encountered, ignore the page rather than failing with Mark Hammond <mhammond@skippinet.com.au> parents: 26162 diff changeset	408 self.format_url(url_pair), msg)
f42fbc7c7d8e When bad HTML is encountered, ignore the page rather than failing with Mark Hammond <mhammond@skippinet.com.au> parents: 26162 diff changeset	409 # Dont actually mark the URL as bad - it exists, just
f42fbc7c7d8e When bad HTML is encountered, ignore the page rather than failing with Mark Hammond <mhammond@skippinet.com.au> parents: 26162 diff changeset	410 # we can't parse it!
f42fbc7c7d8e When bad HTML is encountered, ignore the page rather than failing with Mark Hammond <mhammond@skippinet.com.au> parents: 26162 diff changeset	411 page = None
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	412 if page:
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	413 # Store the page which corresponds to this URL.
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	414 self.name_table[url] = page
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	415 # If there is a fragment in this url_pair, and it's not
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	416 # in the list of names for the page, call setbad(), since
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	417 # it's a missing anchor.
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	418 if local_fragment and local_fragment not in page.getnames():
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	419 self.setbad(url_pair, ("Missing name anchor `%s'" % local_fragment))
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	420 for info in page.getlinkinfos():
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	421 # getlinkinfos() now returns the fragment as well,
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	422 # and we store that fragment here in the "todo" dictionary.
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	423 link, rawlink, fragment = info
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	424 # However, we don't want the fragment as the origin, since
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	425 # the origin is logically a page.
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	426 origin = url, rawlink
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	427 self.newlink((link, fragment), origin)
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	428 else:
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	429 # If no page has been created yet, we want to
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	430 # record that fact.
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	431 self.name_table[url_pair[0]] = None
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	432 self.markdone(url_pair)
5076 4edab474974c Many misc changes. Guido van Rossum <guido@python.org> parents: 5069 diff changeset	433
5086 aa73da661cdf Restructured Checker class to get rid of 'ext' table. Guido van Rossum <guido@python.org> parents: 5083 diff changeset	434 def newlink(self, url, origin):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	435 if self.done.has_key(url):
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	436 self.newdonelink(url, origin)
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	437 else:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	438 self.newtodolink(url, origin)
5076 4edab474974c Many misc changes. Guido van Rossum <guido@python.org> parents: 5069 diff changeset	439
4edab474974c Many misc changes. Guido van Rossum <guido@python.org> parents: 5069 diff changeset	440 def newdonelink(self, url, origin):
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	441 if origin not in self.done[url]:
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	442 self.done[url].append(origin)
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	443
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	444 # Call self.format_url(), since the URL here
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	445 # is now a (URL, fragment) pair.
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	446 self.note(3, " Done link %s", self.format_url(url))
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	447
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	448 # Make sure that if it's bad, that the origin gets added.
11476 f21f19886ee3 Samuel L. Bayer: Guido van Rossum <guido@python.org> parents: 10743 diff changeset	449 if self.bad.has_key(url):
f21f19886ee3 Samuel L. Bayer: Guido van Rossum <guido@python.org> parents: 10743 diff changeset	450 source, rawlink = origin
f21f19886ee3 Samuel L. Bayer: Guido van Rossum <guido@python.org> parents: 10743 diff changeset	451 triple = url, rawlink, self.bad[url]
f21f19886ee3 Samuel L. Bayer: Guido van Rossum <guido@python.org> parents: 10743 diff changeset	452 self.seterror(source, triple)
5076 4edab474974c Many misc changes. Guido van Rossum <guido@python.org> parents: 5069 diff changeset	453
4edab474974c Many misc changes. Guido van Rossum <guido@python.org> parents: 5069 diff changeset	454 def newtodolink(self, url, origin):
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	455 # Call self.format_url(), since the URL here
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	456 # is now a (URL, fragment) pair.
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	457 if self.todo.has_key(url):
11476 f21f19886ee3 Samuel L. Bayer: Guido van Rossum <guido@python.org> parents: 10743 diff changeset	458 if origin not in self.todo[url]:
f21f19886ee3 Samuel L. Bayer: Guido van Rossum <guido@python.org> parents: 10743 diff changeset	459 self.todo[url].append(origin)
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	460 self.note(3, " Seen todo link %s", self.format_url(url))
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	461 else:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	462 self.todo[url] = [origin]
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	463 self.note(3, " New todo link %s", self.format_url(url))
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	464
32736 c3a096bfbd6b Whitespace normalization, via reindent.py. Tim Peters <tim.peters@gmail.com> parents: 31743 diff changeset	465 def format_url(self, url):
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	466 link, fragment = url
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	467 if fragment: return link + "#" + fragment
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	468 else: return link
5076 4edab474974c Many misc changes. Guido van Rossum <guido@python.org> parents: 5069 diff changeset	469
4edab474974c Many misc changes. Guido van Rossum <guido@python.org> parents: 5069 diff changeset	470 def markdone(self, url):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	471 self.done[url] = self.todo[url]
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	472 del self.todo[url]
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	473 self.changed = 1
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	474
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	475 def inroots(self, url):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	476 for root in self.roots:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	477 if url[:len(root)] == root:
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	478 return self.isallowed(root, url)
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	479 return 0
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	480
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	481 def isallowed(self, root, url):
a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	482 root = urlparse.urljoin(root, "/")
a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	483 return self.robots[root].can_fetch(AGENTNAME, url)
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	484
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	485 def getpage(self, url_pair):
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	486 # Incoming argument name is a (URL, fragment) pair.
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	487 # The page may have been cached in the name_table variable.
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	488 url, fragment = url_pair
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	489 if self.name_table.has_key(url):
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	490 return self.name_table[url]
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	491
22219 78bdfbb6e59e I presume most of the fixes currently hitting the tree should go into Michael W. Hudson <mwh@python.net> parents: 21361 diff changeset	492 scheme, path = urllib.splittype(url)
17310 0cdfccdb6871 A number of improvements based on a discussion with Chris McCafferty Fred Drake <fdrake@acm.org> parents: 12072 diff changeset	493 if scheme in ('mailto', 'news', 'javascript', 'telnet'):
0cdfccdb6871 A number of improvements based on a discussion with Chris McCafferty Fred Drake <fdrake@acm.org> parents: 12072 diff changeset	494 self.note(1, " Not checking %s URL" % scheme)
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	495 return None
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	496 isint = self.inroots(url)
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	497
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	498 # Ensure that openpage gets the URL pair to
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	499 # print out its error message and record the error pair
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	500 # correctly.
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	501 if not isint:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	502 if not self.checkext:
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	503 self.note(1, " Not checking ext link")
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	504 return None
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	505 f = self.openpage(url_pair)
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	506 if f:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	507 self.safeclose(f)
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	508 return None
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	509 text, nurl = self.readhtml(url_pair)
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	510
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	511 if nurl != url:
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	512 self.note(1, " Redirected to %s", nurl)
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	513 url = nurl
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	514 if text:
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	515 return Page(text, url, maxpage=self.maxpage, checker=self)
7447 826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	516
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	517 # These next three functions take (URL, fragment) pairs as
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	518 # arguments, so that openpage() receives the appropriate tuple to
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	519 # record error messages.
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	520 def readhtml(self, url_pair):
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	521 url, fragment = url_pair
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	522 text = None
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	523 f, url = self.openhtml(url_pair)
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	524 if f:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	525 text = f.read()
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	526 f.close()
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	527 return text, url
7447 826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	528
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	529 def openhtml(self, url_pair):
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	530 url, fragment = url_pair
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	531 f = self.openpage(url_pair)
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	532 if f:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	533 url = f.geturl()
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	534 info = f.info()
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	535 if not self.checkforhtml(info, url):
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	536 self.safeclose(f)
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	537 f = None
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	538 return f, url
7447 826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	539
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	540 def openpage(self, url_pair):
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	541 url, fragment = url_pair
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	542 try:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	543 return self.urlopener.open(url)
21361 e8b098fe3df8 Fix SF bug #482171: webchecker dies on file: URLs w/o robots.txt Guido van Rossum <guido@python.org> parents: 17318 diff changeset	544 except (OSError, IOError), msg:
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	545 msg = self.sanitize(msg)
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	546 self.note(0, "Error %s", msg)
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	547 if self.verbose > 0:
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	548 self.show(" HREF ", url, " from", self.todo[url_pair])
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	549 self.setbad(url_pair, msg)
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	550 return None
7447 826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	551
826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	552 def checkforhtml(self, info, url):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	553 if info.has_key('content-type'):
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 23704 diff changeset	554 ctype = cgi.parse_header(info['content-type'])[0].lower()
26162 3c083f881655 Handle the Content-Type header a little more appropriately: if it Fred Drake <fdrake@acm.org> parents: 25219 diff changeset	555 if ';' in ctype:
3c083f881655 Handle the Content-Type header a little more appropriately: if it Fred Drake <fdrake@acm.org> parents: 25219 diff changeset	556 # handle content-type: text/html; charset=iso8859-1 :
3c083f881655 Handle the Content-Type header a little more appropriately: if it Fred Drake <fdrake@acm.org> parents: 25219 diff changeset	557 ctype = ctype.split(';', 1)[0].strip()
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	558 else:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	559 if url[-1:] == "/":
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	560 return 1
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	561 ctype, encoding = mimetypes.guess_type(url)
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	562 if ctype == 'text/html':
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	563 return 1
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	564 else:
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	565 self.note(1, " Not HTML, mime type %s", ctype)
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	566 return 0
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	567
5076 4edab474974c Many misc changes. Guido van Rossum <guido@python.org> parents: 5069 diff changeset	568 def setgood(self, url):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	569 if self.bad.has_key(url):
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	570 del self.bad[url]
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	571 self.changed = 1
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	572 self.note(0, "(Clear previously seen error)")
5076 4edab474974c Many misc changes. Guido van Rossum <guido@python.org> parents: 5069 diff changeset	573
4edab474974c Many misc changes. Guido van Rossum <guido@python.org> parents: 5069 diff changeset	574 def setbad(self, url, msg):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	575 if self.bad.has_key(url) and self.bad[url] == msg:
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	576 self.note(0, "(Seen this error before)")
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	577 return
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	578 self.bad[url] = msg
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	579 self.changed = 1
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	580 self.markerror(url)
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	581
5086 aa73da661cdf Restructured Checker class to get rid of 'ext' table. Guido van Rossum <guido@python.org> parents: 5083 diff changeset	582 def markerror(self, url):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	583 try:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	584 origins = self.todo[url]
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	585 except KeyError:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	586 origins = self.done[url]
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	587 for source, rawlink in origins:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	588 triple = url, rawlink, self.bad[url]
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	589 self.seterror(source, triple)
5086 aa73da661cdf Restructured Checker class to get rid of 'ext' table. Guido van Rossum <guido@python.org> parents: 5083 diff changeset	590
aa73da661cdf Restructured Checker class to get rid of 'ext' table. Guido van Rossum <guido@python.org> parents: 5083 diff changeset	591 def seterror(self, url, triple):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	592 try:
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	593 # Because of the way the URLs are now processed, I need to
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	594 # check to make sure the URL hasn't been entered in the
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	595 # error list. The first element of the triple here is a
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	596 # (URL, fragment) pair, but the URL key is not, since it's
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	597 # from the list of origins.
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	598 if triple not in self.errors[url]:
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	599 self.errors[url].append(triple)
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	600 except KeyError:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	601 self.errors[url] = [triple]
5076 4edab474974c Many misc changes. Guido van Rossum <guido@python.org> parents: 5069 diff changeset	602
7447 826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	603 # The following used to be toplevel functions; they have been
826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	604 # changed into methods so they can be overridden in subclasses.
826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	605
826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	606 def show(self, p1, link, p2, origins):
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	607 self.message("%s %s", p1, link)
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	608 i = 0
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	609 for source, rawlink in origins:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	610 i = i+1
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	611 if i == 2:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	612 p2 = ' '*len(p2)
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	613 if rawlink != link: s = " (%s)" % rawlink
a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	614 else: s = ""
a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	615 self.message("%s %s%s", p2, source, s)
7447 826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	616
826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	617 def sanitize(self, msg):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	618 if isinstance(IOError, ClassType) and isinstance(msg, IOError):
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	619 # Do the other branch recursively
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	620 msg.args = self.sanitize(msg.args)
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	621 elif isinstance(msg, TupleType):
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	622 if len(msg) >= 4 and msg[0] == 'http error' and \
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	623 isinstance(msg[3], InstanceType):
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	624 # Remove the Message instance -- it may contain
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	625 # a file object which prevents pickling.
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	626 msg = msg[:3] + msg[4:]
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	627 return msg
7447 826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	628
826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	629 def safeclose(self, f):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	630 try:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	631 url = f.geturl()
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	632 except AttributeError:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	633 pass
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	634 else:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	635 if url[:4] == 'ftp:' or url[:7] == 'file://':
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	636 # Apparently ftp connections don't like to be closed
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	637 # prematurely...
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	638 text = f.read()
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	639 f.close()
7447 826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	640
826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	641 def save_pickle(self, dumpfile=DUMPFILE):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	642 if not self.changed:
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	643 self.note(0, "\nNo need to save checkpoint")
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	644 elif not dumpfile:
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	645 self.note(0, "No dumpfile, won't save checkpoint")
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	646 else:
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	647 self.note(0, "\nSaving checkpoint to %s ...", dumpfile)
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	648 newfile = dumpfile + ".new"
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	649 f = open(newfile, "wb")
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	650 pickle.dump(self, f)
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	651 f.close()
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	652 try:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	653 os.unlink(dumpfile)
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	654 except os.error:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	655 pass
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	656 os.rename(newfile, dumpfile)
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	657 self.note(0, "Done.")
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	658 return 1
7447 826c71801911 Major overhaul. Don't use global variable (e.g. verbose); use Guido van Rossum <guido@python.org> parents: 6398 diff changeset	659
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	660
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	661 class Page:
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	662
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	663 def __init__(self, text, url, verbose=VERBOSE, maxpage=MAXPAGE, checker=None):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	664 self.text = text
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	665 self.url = url
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	666 self.verbose = verbose
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	667 self.maxpage = maxpage
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	668 self.checker = checker
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	669
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	670 # The parsing of the page is done in the __init__() routine in
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	671 # order to initialize the list of names the file
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	672 # contains. Stored the parser in an instance variable. Passed
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	673 # the URL to MyHTMLParser().
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	674 size = len(self.text)
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	675 if size > self.maxpage:
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	676 self.note(0, "Skip huge file %s (%.0f Kbytes)", self.url, (size*0.001))
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	677 self.parser = None
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	678 return
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	679 self.checker.note(2, " Parsing %s (%d bytes)", self.url, size)
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	680 self.parser = MyHTMLParser(url, verbose=self.verbose,
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	681 checker=self.checker)
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	682 self.parser.feed(self.text)
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	683 self.parser.close()
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	684
8857 f7d4a585da01 Added note() message to Page class -- this was used but didn't exist. Guido van Rossum <guido@python.org> parents: 8647 diff changeset	685 def note(self, level, msg, *args):
f7d4a585da01 Added note() message to Page class -- this was used but didn't exist. Guido van Rossum <guido@python.org> parents: 8647 diff changeset	686 if self.checker:
f7d4a585da01 Added note() message to Page class -- this was used but didn't exist. Guido van Rossum <guido@python.org> parents: 8647 diff changeset	687 apply(self.checker.note, (level, msg) + args)
f7d4a585da01 Added note() message to Page class -- this was used but didn't exist. Guido van Rossum <guido@python.org> parents: 8647 diff changeset	688 else:
f7d4a585da01 Added note() message to Page class -- this was used but didn't exist. Guido van Rossum <guido@python.org> parents: 8647 diff changeset	689 if self.verbose >= level:
f7d4a585da01 Added note() message to Page class -- this was used but didn't exist. Guido van Rossum <guido@python.org> parents: 8647 diff changeset	690 if args:
f7d4a585da01 Added note() message to Page class -- this was used but didn't exist. Guido van Rossum <guido@python.org> parents: 8647 diff changeset	691 msg = msg%args
f7d4a585da01 Added note() message to Page class -- this was used but didn't exist. Guido van Rossum <guido@python.org> parents: 8647 diff changeset	692 print msg
f7d4a585da01 Added note() message to Page class -- this was used but didn't exist. Guido van Rossum <guido@python.org> parents: 8647 diff changeset	693
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	694 # Method to retrieve names.
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	695 def getnames(self):
12072 67d4128da254 Fix suggested by Magnus Kessler: in class Page, it is possible for Guido van Rossum <guido@python.org> parents: 11481 diff changeset	696 if self.parser:
67d4128da254 Fix suggested by Magnus Kessler: in class Page, it is possible for Guido van Rossum <guido@python.org> parents: 11481 diff changeset	697 return self.parser.names
67d4128da254 Fix suggested by Magnus Kessler: in class Page, it is possible for Guido van Rossum <guido@python.org> parents: 11481 diff changeset	698 else:
67d4128da254 Fix suggested by Magnus Kessler: in class Page, it is possible for Guido van Rossum <guido@python.org> parents: 11481 diff changeset	699 return []
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	700
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	701 def getlinkinfos(self):
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	702 # File reading is done in __init__() routine. Store parser in
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	703 # local variable to indicate success of parsing.
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	704
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	705 # If no parser was stored, fail.
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	706 if not self.parser: return []
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	707
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	708 rawlinks = self.parser.getlinks()
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	709 base = urlparse.urljoin(self.url, self.parser.getbase() or "")
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	710 infos = []
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	711 for rawlink in rawlinks:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	712 t = urlparse.urlparse(rawlink)
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	713 # DON'T DISCARD THE FRAGMENT! Instead, include
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	714 # it in the tuples which are returned. See Checker.dopage().
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	715 fragment = t[-1]
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	716 t = t[:-1] + ('',)
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	717 rawlink = urlparse.urlunparse(t)
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	718 link = urlparse.urljoin(base, rawlink)
32736 c3a096bfbd6b Whitespace normalization, via reindent.py. Tim Peters <tim.peters@gmail.com> parents: 31743 diff changeset	719 infos.append((link, rawlink, fragment))
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	720
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	721 return infos
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	722
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	723
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	724 class MyStringIO(StringIO.StringIO):
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	725
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	726 def __init__(self, url, info):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	727 self.__url = url
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	728 self.__info = info
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	729 StringIO.StringIO.__init__(self)
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	730
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	731 def info(self):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	732 return self.__info
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	733
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	734 def geturl(self):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	735 return self.__url
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	736
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	737
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	738 class MyURLopener(urllib.FancyURLopener):
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	739
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	740 http_error_default = urllib.URLopener.http_error_default
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	741
5069 fa31476a2786 Set proper User-agent header (Python-webchecker/<version>). Guido van Rossum <guido@python.org> parents: 5068 diff changeset	742 def __init__(*args):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	743 self = args[0]
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	744 apply(urllib.FancyURLopener.__init__, args)
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	745 self.addheaders = [
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	746 ('User-agent', 'Python-webchecker/%s' % __version__),
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	747 ]
5423 bccb8d742964 Avoid the fancy handler for error 401 (request authentication). Guido van Rossum <guido@python.org> parents: 5086 diff changeset	748
bccb8d742964 Avoid the fancy handler for error 401 (request authentication). Guido van Rossum <guido@python.org> parents: 5086 diff changeset	749 def http_error_401(self, url, fp, errcode, errmsg, headers):
bccb8d742964 Avoid the fancy handler for error 401 (request authentication). Guido van Rossum <guido@python.org> parents: 5086 diff changeset	750 return None
5069 fa31476a2786 Set proper User-agent header (Python-webchecker/<version>). Guido van Rossum <guido@python.org> parents: 5068 diff changeset	751
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	752 def open_file(self, url):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	753 path = urllib.url2pathname(urllib.unquote(url))
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	754 if os.path.isdir(path):
10743 855887a2cc60 Some changes (maybe not enough?) to make it work on Windows with local Guido van Rossum <guido@python.org> parents: 8857 diff changeset	755 if path[-1] != os.sep:
855887a2cc60 Some changes (maybe not enough?) to make it work on Windows with local Guido van Rossum <guido@python.org> parents: 8857 diff changeset	756 url = url + '/'
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	757 indexpath = os.path.join(path, "index.html")
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	758 if os.path.exists(indexpath):
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	759 return self.open_file(url + "index.html")
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	760 try:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	761 names = os.listdir(path)
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	762 except os.error, msg:
39197 b81642e52287 Use sys.exc_info() Andrew M. Kuchling <amk@amk.ca> parents: 38627 diff changeset	763 exc_type, exc_value, exc_tb = sys.exc_info()
b81642e52287 Use sys.exc_info() Andrew M. Kuchling <amk@amk.ca> parents: 38627 diff changeset	764 raise IOError, msg, exc_tb
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	765 names.sort()
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	766 s = MyStringIO("file:"+url, {'content-type': 'text/html'})
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	767 s.write('<BASE HREF="file:%s">\n' %
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	768 urllib.quote(os.path.join(path, "")))
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	769 for name in names:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	770 q = urllib.quote(name)
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	771 s.write('<A HREF="%s">%s</A>\n' % (q, q))
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	772 s.seek(0)
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	773 return s
10743 855887a2cc60 Some changes (maybe not enough?) to make it work on Windows with local Guido van Rossum <guido@python.org> parents: 8857 diff changeset	774 return urllib.FancyURLopener.open_file(self, url)
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	775
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	776
5076 4edab474974c Many misc changes. Guido van Rossum <guido@python.org> parents: 5069 diff changeset	777 class MyHTMLParser(sgmllib.SGMLParser):
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	778
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	779 def __init__(self, url, verbose=VERBOSE, checker=None):
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	780 self.myverbose = verbose # now unused
a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	781 self.checker = checker
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	782 self.base = None
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	783 self.links = {}
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	784 self.names = []
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	785 self.url = url
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	786 sgmllib.SGMLParser.__init__(self)
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	787
38627 98e81af40007 Normalized a few cases of whitespace in function declarations. Martin Blais <blais@furius.ca> parents: 32736 diff changeset	788 def check_name_id(self, attributes):
31743 907450e4bc67 [Patch #918212] Support XHTML's 'id' attribute, which can be on any element. Andrew M. Kuchling <amk@amk.ca> parents: 28085 diff changeset	789 """ Check the name or id attributes on an element.
907450e4bc67 [Patch #918212] Support XHTML's 'id' attribute, which can be on any element. Andrew M. Kuchling <amk@amk.ca> parents: 28085 diff changeset	790 """
907450e4bc67 [Patch #918212] Support XHTML's 'id' attribute, which can be on any element. Andrew M. Kuchling <amk@amk.ca> parents: 28085 diff changeset	791 # We must rescue the NAME or id (name is deprecated in XHTML)
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	792 # attributes from the anchor, in order to
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	793 # cache the internal anchors which are made
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	794 # available in the page.
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	795 for name, value in attributes:
31743 907450e4bc67 [Patch #918212] Support XHTML's 'id' attribute, which can be on any element. Andrew M. Kuchling <amk@amk.ca> parents: 28085 diff changeset	796 if name == "name" or name == "id":
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	797 if value in self.names:
31743 907450e4bc67 [Patch #918212] Support XHTML's 'id' attribute, which can be on any element. Andrew M. Kuchling <amk@amk.ca> parents: 28085 diff changeset	798 self.checker.message("WARNING: duplicate ID name %s in %s",
11481 7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	799 value, self.url)
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	800 else: self.names.append(value)
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	801 break
7dc927204d65 Integrated Sam Bayer's wcnew.py code. It seems silly to keep two Guido van Rossum <guido@python.org> parents: 11476 diff changeset	802
38627 98e81af40007 Normalized a few cases of whitespace in function declarations. Martin Blais <blais@furius.ca> parents: 32736 diff changeset	803 def unknown_starttag(self, tag, attributes):
31743 907450e4bc67 [Patch #918212] Support XHTML's 'id' attribute, which can be on any element. Andrew M. Kuchling <amk@amk.ca> parents: 28085 diff changeset	804 """ In XHTML, you can have id attributes on any element.
907450e4bc67 [Patch #918212] Support XHTML's 'id' attribute, which can be on any element. Andrew M. Kuchling <amk@amk.ca> parents: 28085 diff changeset	805 """
907450e4bc67 [Patch #918212] Support XHTML's 'id' attribute, which can be on any element. Andrew M. Kuchling <amk@amk.ca> parents: 28085 diff changeset	806 self.check_name_id(attributes)
907450e4bc67 [Patch #918212] Support XHTML's 'id' attribute, which can be on any element. Andrew M. Kuchling <amk@amk.ca> parents: 28085 diff changeset	807
907450e4bc67 [Patch #918212] Support XHTML's 'id' attribute, which can be on any element. Andrew M. Kuchling <amk@amk.ca> parents: 28085 diff changeset	808 def start_a(self, attributes):
907450e4bc67 [Patch #918212] Support XHTML's 'id' attribute, which can be on any element. Andrew M. Kuchling <amk@amk.ca> parents: 28085 diff changeset	809 self.link_attr(attributes, 'href')
907450e4bc67 [Patch #918212] Support XHTML's 'id' attribute, which can be on any element. Andrew M. Kuchling <amk@amk.ca> parents: 28085 diff changeset	810 self.check_name_id(attributes)
907450e4bc67 [Patch #918212] Support XHTML's 'id' attribute, which can be on any element. Andrew M. Kuchling <amk@amk.ca> parents: 28085 diff changeset	811
5083 7ab1aa08d348 Process <img> and <frame> tags. Don't bother skipping second href. Guido van Rossum <guido@python.org> parents: 5080 diff changeset	812 def end_a(self): pass
7ab1aa08d348 Process <img> and <frame> tags. Don't bother skipping second href. Guido van Rossum <guido@python.org> parents: 5080 diff changeset	813
6398 9ed9eb4a7f23 Several changes: Guido van Rossum <guido@python.org> parents: 5423 diff changeset	814 def do_area(self, attributes):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	815 self.link_attr(attributes, 'href')
31743 907450e4bc67 [Patch #918212] Support XHTML's 'id' attribute, which can be on any element. Andrew M. Kuchling <amk@amk.ca> parents: 28085 diff changeset	816 self.check_name_id(attributes)
6398 9ed9eb4a7f23 Several changes: Guido van Rossum <guido@python.org> parents: 5423 diff changeset	817
17310 0cdfccdb6871 A number of improvements based on a discussion with Chris McCafferty Fred Drake <fdrake@acm.org> parents: 12072 diff changeset	818 def do_body(self, attributes):
17318 b25a2413020e Added more link attributes based on additonal information from Chris Fred Drake <fdrake@acm.org> parents: 17310 diff changeset	819 self.link_attr(attributes, 'background', 'bgsound')
31743 907450e4bc67 [Patch #918212] Support XHTML's 'id' attribute, which can be on any element. Andrew M. Kuchling <amk@amk.ca> parents: 28085 diff changeset	820 self.check_name_id(attributes)
17310 0cdfccdb6871 A number of improvements based on a discussion with Chris McCafferty Fred Drake <fdrake@acm.org> parents: 12072 diff changeset	821
5083 7ab1aa08d348 Process <img> and <frame> tags. Don't bother skipping second href. Guido van Rossum <guido@python.org> parents: 5080 diff changeset	822 def do_img(self, attributes):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	823 self.link_attr(attributes, 'src', 'lowsrc')
31743 907450e4bc67 [Patch #918212] Support XHTML's 'id' attribute, which can be on any element. Andrew M. Kuchling <amk@amk.ca> parents: 28085 diff changeset	824 self.check_name_id(attributes)
5083 7ab1aa08d348 Process <img> and <frame> tags. Don't bother skipping second href. Guido van Rossum <guido@python.org> parents: 5080 diff changeset	825
7ab1aa08d348 Process <img> and <frame> tags. Don't bother skipping second href. Guido van Rossum <guido@python.org> parents: 5080 diff changeset	826 def do_frame(self, attributes):
17310 0cdfccdb6871 A number of improvements based on a discussion with Chris McCafferty Fred Drake <fdrake@acm.org> parents: 12072 diff changeset	827 self.link_attr(attributes, 'src', 'longdesc')
31743 907450e4bc67 [Patch #918212] Support XHTML's 'id' attribute, which can be on any element. Andrew M. Kuchling <amk@amk.ca> parents: 28085 diff changeset	828 self.check_name_id(attributes)
17310 0cdfccdb6871 A number of improvements based on a discussion with Chris McCafferty Fred Drake <fdrake@acm.org> parents: 12072 diff changeset	829
0cdfccdb6871 A number of improvements based on a discussion with Chris McCafferty Fred Drake <fdrake@acm.org> parents: 12072 diff changeset	830 def do_iframe(self, attributes):
0cdfccdb6871 A number of improvements based on a discussion with Chris McCafferty Fred Drake <fdrake@acm.org> parents: 12072 diff changeset	831 self.link_attr(attributes, 'src', 'longdesc')
31743 907450e4bc67 [Patch #918212] Support XHTML's 'id' attribute, which can be on any element. Andrew M. Kuchling <amk@amk.ca> parents: 28085 diff changeset	832 self.check_name_id(attributes)
17310 0cdfccdb6871 A number of improvements based on a discussion with Chris McCafferty Fred Drake <fdrake@acm.org> parents: 12072 diff changeset	833
0cdfccdb6871 A number of improvements based on a discussion with Chris McCafferty Fred Drake <fdrake@acm.org> parents: 12072 diff changeset	834 def do_link(self, attributes):
0cdfccdb6871 A number of improvements based on a discussion with Chris McCafferty Fred Drake <fdrake@acm.org> parents: 12072 diff changeset	835 for name, value in attributes:
0cdfccdb6871 A number of improvements based on a discussion with Chris McCafferty Fred Drake <fdrake@acm.org> parents: 12072 diff changeset	836 if name == "rel":
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 23704 diff changeset	837 parts = value.lower().split()
17310 0cdfccdb6871 A number of improvements based on a discussion with Chris McCafferty Fred Drake <fdrake@acm.org> parents: 12072 diff changeset	838 if ( parts == ["stylesheet"]
0cdfccdb6871 A number of improvements based on a discussion with Chris McCafferty Fred Drake <fdrake@acm.org> parents: 12072 diff changeset	839 or parts == ["alternate", "stylesheet"]):
0cdfccdb6871 A number of improvements based on a discussion with Chris McCafferty Fred Drake <fdrake@acm.org> parents: 12072 diff changeset	840 self.link_attr(attributes, "href")
0cdfccdb6871 A number of improvements based on a discussion with Chris McCafferty Fred Drake <fdrake@acm.org> parents: 12072 diff changeset	841 break
31743 907450e4bc67 [Patch #918212] Support XHTML's 'id' attribute, which can be on any element. Andrew M. Kuchling <amk@amk.ca> parents: 28085 diff changeset	842 self.check_name_id(attributes)
17310 0cdfccdb6871 A number of improvements based on a discussion with Chris McCafferty Fred Drake <fdrake@acm.org> parents: 12072 diff changeset	843
0cdfccdb6871 A number of improvements based on a discussion with Chris McCafferty Fred Drake <fdrake@acm.org> parents: 12072 diff changeset	844 def do_object(self, attributes):
0cdfccdb6871 A number of improvements based on a discussion with Chris McCafferty Fred Drake <fdrake@acm.org> parents: 12072 diff changeset	845 self.link_attr(attributes, 'data', 'usemap')
31743 907450e4bc67 [Patch #918212] Support XHTML's 'id' attribute, which can be on any element. Andrew M. Kuchling <amk@amk.ca> parents: 28085 diff changeset	846 self.check_name_id(attributes)
17310 0cdfccdb6871 A number of improvements based on a discussion with Chris McCafferty Fred Drake <fdrake@acm.org> parents: 12072 diff changeset	847
0cdfccdb6871 A number of improvements based on a discussion with Chris McCafferty Fred Drake <fdrake@acm.org> parents: 12072 diff changeset	848 def do_script(self, attributes):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	849 self.link_attr(attributes, 'src')
31743 907450e4bc67 [Patch #918212] Support XHTML's 'id' attribute, which can be on any element. Andrew M. Kuchling <amk@amk.ca> parents: 28085 diff changeset	850 self.check_name_id(attributes)
5083 7ab1aa08d348 Process <img> and <frame> tags. Don't bother skipping second href. Guido van Rossum <guido@python.org> parents: 5080 diff changeset	851
17318 b25a2413020e Added more link attributes based on additonal information from Chris Fred Drake <fdrake@acm.org> parents: 17310 diff changeset	852 def do_table(self, attributes):
b25a2413020e Added more link attributes based on additonal information from Chris Fred Drake <fdrake@acm.org> parents: 17310 diff changeset	853 self.link_attr(attributes, 'background')
31743 907450e4bc67 [Patch #918212] Support XHTML's 'id' attribute, which can be on any element. Andrew M. Kuchling <amk@amk.ca> parents: 28085 diff changeset	854 self.check_name_id(attributes)
17318 b25a2413020e Added more link attributes based on additonal information from Chris Fred Drake <fdrake@acm.org> parents: 17310 diff changeset	855
b25a2413020e Added more link attributes based on additonal information from Chris Fred Drake <fdrake@acm.org> parents: 17310 diff changeset	856 def do_td(self, attributes):
b25a2413020e Added more link attributes based on additonal information from Chris Fred Drake <fdrake@acm.org> parents: 17310 diff changeset	857 self.link_attr(attributes, 'background')
31743 907450e4bc67 [Patch #918212] Support XHTML's 'id' attribute, which can be on any element. Andrew M. Kuchling <amk@amk.ca> parents: 28085 diff changeset	858 self.check_name_id(attributes)
17318 b25a2413020e Added more link attributes based on additonal information from Chris Fred Drake <fdrake@acm.org> parents: 17310 diff changeset	859
b25a2413020e Added more link attributes based on additonal information from Chris Fred Drake <fdrake@acm.org> parents: 17310 diff changeset	860 def do_th(self, attributes):
b25a2413020e Added more link attributes based on additonal information from Chris Fred Drake <fdrake@acm.org> parents: 17310 diff changeset	861 self.link_attr(attributes, 'background')
31743 907450e4bc67 [Patch #918212] Support XHTML's 'id' attribute, which can be on any element. Andrew M. Kuchling <amk@amk.ca> parents: 28085 diff changeset	862 self.check_name_id(attributes)
17318 b25a2413020e Added more link attributes based on additonal information from Chris Fred Drake <fdrake@acm.org> parents: 17310 diff changeset	863
b25a2413020e Added more link attributes based on additonal information from Chris Fred Drake <fdrake@acm.org> parents: 17310 diff changeset	864 def do_tr(self, attributes):
b25a2413020e Added more link attributes based on additonal information from Chris Fred Drake <fdrake@acm.org> parents: 17310 diff changeset	865 self.link_attr(attributes, 'background')
31743 907450e4bc67 [Patch #918212] Support XHTML's 'id' attribute, which can be on any element. Andrew M. Kuchling <amk@amk.ca> parents: 28085 diff changeset	866 self.check_name_id(attributes)
17318 b25a2413020e Added more link attributes based on additonal information from Chris Fred Drake <fdrake@acm.org> parents: 17310 diff changeset	867
5083 7ab1aa08d348 Process <img> and <frame> tags. Don't bother skipping second href. Guido van Rossum <guido@python.org> parents: 5080 diff changeset	868 def link_attr(self, attributes, *args):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	869 for name, value in attributes:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	870 if name in args:
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 23704 diff changeset	871 if value: value = value.strip()
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	872 if value: self.links[value] = None
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	873
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	874 def do_base(self, attributes):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	875 for name, value in attributes:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	876 if name == 'href':
25219 8a7a2f1f88f0 Apply diff2.txt from SF patch http://www.python.org/sf/572113 Walter Dörwald <walter@livinglogic.de> parents: 23704 diff changeset	877 if value: value = value.strip()
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	878 if value:
8647 a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	879 if self.checker:
a2beaedf0aee Instead of printint, use self.message() or self.note(). Guido van Rossum <guido@python.org> parents: 8501 diff changeset	880 self.checker.note(1, " Base %s", value)
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	881 self.base = value
31743 907450e4bc67 [Patch #918212] Support XHTML's 'id' attribute, which can be on any element. Andrew M. Kuchling <amk@amk.ca> parents: 28085 diff changeset	882 self.check_name_id(attributes)
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	883
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	884 def getlinks(self):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	885 return self.links.keys()
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	886
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	887 def getbase(self):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 7447 diff changeset	888 return self.base
5062 d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	889
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	890
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	891 if __name__ == '__main__':
d681631d992f web tree checker Guido van Rossum <guido@python.org> parents: diff changeset	892 main()

Mercurial > cpython-lru_cache-weakref

annotate Tools/webchecker/webchecker.py @ 54294:dba767bc3155 legacy-trunk