cpython-lru_cache-weakref: Lib/robotparser.py annotate

annotate Lib/robotparser.py @ 25838:8540887bcec6 2.2

Backport of 2.183: Fix memory leak in add_subclass() found by NealN with valgrind.

author	Guido van Rossum <guido@python.org>
date	Fri, 18 Oct 2002 13:42:21 +0000
parents	c0a16b4c9edb
children	054d9bc0d860 736fb5aec9b3

rev	line source
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	1 """ robotparser.py
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	2
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	3 Copyright (C) 2000 Bastian Kleineidam
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	4
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	5 You can choose between two licenses when using this package:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	6 1) GNU GPLv2
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	7 2) PYTHON 2.0 OPEN SOURCE LICENSE
5063 f0731fa4447f Skip Montanaro's robots.txt parser. Guido van Rossum <guido@python.org> parents: diff changeset	8
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	9 The robots.txt Exclusion Protocol is implemented as specified in
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	10 http://info.webcrawler.com/mak/projects/robots/norobots-rfc.html
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	11 """
16494 f4d15fdd15dd String method conversion. Eric S. Raymond <esr@thyrsus.com> parents: 16074 diff changeset	12 import re,urlparse,urllib
5063 f0731fa4447f Skip Montanaro's robots.txt parser. Guido van Rossum <guido@python.org> parents: diff changeset	13
16070 b5345e99596b added __all__ lists to a number of Python modules Skip Montanaro <skip@pobox.com> parents: 16067 diff changeset	14 __all__ = ["RobotFileParser"]
b5345e99596b added __all__ lists to a number of Python modules Skip Montanaro <skip@pobox.com> parents: 16067 diff changeset	15
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	16 debug = 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	17
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	18 def _debug(msg):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	19 if debug: print msg
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	20
5063 f0731fa4447f Skip Montanaro's robots.txt parser. Guido van Rossum <guido@python.org> parents: diff changeset	21
f0731fa4447f Skip Montanaro's robots.txt parser. Guido van Rossum <guido@python.org> parents: diff changeset	22 class RobotFileParser:
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	23 def __init__(self, url=''):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	24 self.entries = []
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	25 self.disallow_all = 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	26 self.allow_all = 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	27 self.set_url(url)
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 5063 diff changeset	28 self.last_checked = 0
5063 f0731fa4447f Skip Montanaro's robots.txt parser. Guido van Rossum <guido@python.org> parents: diff changeset	29
f0731fa4447f Skip Montanaro's robots.txt parser. Guido van Rossum <guido@python.org> parents: diff changeset	30 def mtime(self):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 5063 diff changeset	31 return self.last_checked
5063 f0731fa4447f Skip Montanaro's robots.txt parser. Guido van Rossum <guido@python.org> parents: diff changeset	32
f0731fa4447f Skip Montanaro's robots.txt parser. Guido van Rossum <guido@python.org> parents: diff changeset	33 def modified(self):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 5063 diff changeset	34 import time
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 5063 diff changeset	35 self.last_checked = time.time()
5063 f0731fa4447f Skip Montanaro's robots.txt parser. Guido van Rossum <guido@python.org> parents: diff changeset	36
f0731fa4447f Skip Montanaro's robots.txt parser. Guido van Rossum <guido@python.org> parents: diff changeset	37 def set_url(self, url):
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 5063 diff changeset	38 self.url = url
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	39 self.host, self.path = urlparse.urlparse(url)[1:3]
5063 f0731fa4447f Skip Montanaro's robots.txt parser. Guido van Rossum <guido@python.org> parents: diff changeset	40
f0731fa4447f Skip Montanaro's robots.txt parser. Guido van Rossum <guido@python.org> parents: diff changeset	41 def read(self):
16579 a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	42 opener = URLopener()
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	43 f = opener.open(self.url)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	44 lines = f.readlines()
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	45 self.errcode = opener.errcode
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	46 if self.errcode == 401 or self.errcode == 403:
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	47 self.disallow_all = 1
16579 a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	48 _debug("disallow all")
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	49 elif self.errcode >= 400:
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	50 self.allow_all = 1
16579 a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	51 _debug("allow all")
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	52 elif self.errcode == 200 and lines:
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	53 _debug("parse lines")
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	54 self.parse(lines)
5063 f0731fa4447f Skip Montanaro's robots.txt parser. Guido van Rossum <guido@python.org> parents: diff changeset	55
f0731fa4447f Skip Montanaro's robots.txt parser. Guido van Rossum <guido@python.org> parents: diff changeset	56 def parse(self, lines):
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	57 """parse the input lines from a robot.txt file.
16074 7e55fa93f4db Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	58 We allow that a user-agent: line is not preceded by
7e55fa93f4db Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	59 one or more blank lines."""
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	60 state = 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	61 linenumber = 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	62 entry = Entry()
16074 7e55fa93f4db Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	63
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 5063 diff changeset	64 for line in lines:
16494 f4d15fdd15dd String method conversion. Eric S. Raymond <esr@thyrsus.com> parents: 16074 diff changeset	65 line = line.strip()
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	66 linenumber = linenumber + 1
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	67 if not line:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	68 if state==1:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	69 _debug("line %d: warning: you should insert"
16074 7e55fa93f4db Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	70 " allow: or disallow: directives below any"
7e55fa93f4db Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	71 " user-agent: line" % linenumber)
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	72 entry = Entry()
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	73 state = 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	74 elif state==2:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	75 self.entries.append(entry)
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	76 entry = Entry()
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	77 state = 0
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 5063 diff changeset	78 # remove optional comment and strip line
16494 f4d15fdd15dd String method conversion. Eric S. Raymond <esr@thyrsus.com> parents: 16074 diff changeset	79 i = line.find('#')
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	80 if i>=0:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	81 line = line[:i]
16494 f4d15fdd15dd String method conversion. Eric S. Raymond <esr@thyrsus.com> parents: 16074 diff changeset	82 line = line.strip()
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 5063 diff changeset	83 if not line:
efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 5063 diff changeset	84 continue
16494 f4d15fdd15dd String method conversion. Eric S. Raymond <esr@thyrsus.com> parents: 16074 diff changeset	85 line = line.split(':', 1)
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 5063 diff changeset	86 if len(line) == 2:
16494 f4d15fdd15dd String method conversion. Eric S. Raymond <esr@thyrsus.com> parents: 16074 diff changeset	87 line[0] = line[0].strip().lower()
f4d15fdd15dd String method conversion. Eric S. Raymond <esr@thyrsus.com> parents: 16074 diff changeset	88 line[1] = line[1].strip()
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	89 if line[0] == "user-agent":
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	90 if state==2:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	91 _debug("line %d: warning: you should insert a blank"
16074 7e55fa93f4db Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	92 " line before any user-agent"
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	93 " directive" % linenumber)
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	94 self.entries.append(entry)
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	95 entry = Entry()
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	96 entry.useragents.append(line[1])
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	97 state = 1
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	98 elif line[0] == "disallow":
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	99 if state==0:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	100 _debug("line %d: error: you must insert a user-agent:"
16074 7e55fa93f4db Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	101 " directive before this line" % linenumber)
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 5063 diff changeset	102 else:
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	103 entry.rulelines.append(RuleLine(line[1], 0))
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	104 state = 2
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	105 elif line[0] == "allow":
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	106 if state==0:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	107 _debug("line %d: error: you must insert a user-agent:"
16074 7e55fa93f4db Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	108 " directive before this line" % linenumber)
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	109 else:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	110 entry.rulelines.append(RuleLine(line[1], 1))
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 5063 diff changeset	111 else:
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	112 _debug("line %d: warning: unknown key %s" % (linenumber,
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	113 line[0]))
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	114 else:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	115 _debug("line %d: error: malformed line %s"%(linenumber, line))
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	116 if state==2:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	117 self.entries.append(entry)
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	118 _debug("Parsed rules:\n%s" % str(self))
5063 f0731fa4447f Skip Montanaro's robots.txt parser. Guido van Rossum <guido@python.org> parents: diff changeset	119
f0731fa4447f Skip Montanaro's robots.txt parser. Guido van Rossum <guido@python.org> parents: diff changeset	120
12062 c43b38079228 Skip Montanaro: Guido van Rossum <guido@python.org> parents: 7887 diff changeset	121 def can_fetch(self, useragent, url):
c43b38079228 Skip Montanaro: Guido van Rossum <guido@python.org> parents: 7887 diff changeset	122 """using the parsed robots.txt decide if useragent can fetch url"""
16579 a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	123 _debug("Checking robot.txt allowance for:\n user agent: %s\n url: %s" %
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	124 (useragent, url))
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	125 if self.disallow_all:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	126 return 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	127 if self.allow_all:
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 5063 diff changeset	128 return 1
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	129 # search for given user agent matches
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	130 # the first match counts
16579 a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	131 url = urllib.quote(urlparse.urlparse(url)[2]) or "/"
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	132 for entry in self.entries:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	133 if entry.applies_to(useragent):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	134 return entry.allowance(url)
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	135 # agent not found ==> access granted
7887 efbb7e57e6f8 Give in to tabnanny Guido van Rossum <guido@python.org> parents: 5063 diff changeset	136 return 1
5063 f0731fa4447f Skip Montanaro's robots.txt parser. Guido van Rossum <guido@python.org> parents: diff changeset	137
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	138
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	139 def __str__(self):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	140 ret = ""
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	141 for entry in self.entries:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	142 ret = ret + str(entry) + "\n"
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	143 return ret
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	144
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	145
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	146 class RuleLine:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	147 """A rule line is a single "Allow:" (allowance==1) or "Disallow:"
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	148 (allowance==0) followed by a path."""
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	149 def __init__(self, path, allowance):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	150 self.path = urllib.quote(path)
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	151 self.allowance = allowance
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	152
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	153 def applies_to(self, filename):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	154 return self.path=="*" or re.match(self.path, filename)
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	155
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	156 def __str__(self):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	157 return (self.allowance and "Allow" or "Disallow")+": "+self.path
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	158
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	159
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	160 class Entry:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	161 """An entry has one or more user-agents and zero or more rulelines"""
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	162 def __init__(self):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	163 self.useragents = []
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	164 self.rulelines = []
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	165
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	166 def __str__(self):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	167 ret = ""
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	168 for agent in self.useragents:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	169 ret = ret + "User-agent: "+agent+"\n"
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	170 for line in self.rulelines:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	171 ret = ret + str(line) + "\n"
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	172 return ret
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	173
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	174 def applies_to(self, useragent):
16579 a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	175 """check if this entry applies to the specified agent"""
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	176 # split the name token and make it lower case
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	177 useragent = useragent.split("/")[0].lower()
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	178 for agent in self.useragents:
16579 a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	179 if agent=='*':
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	180 # we have the catch-all agent
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	181 return 1
16579 a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	182 agent = agent.lower()
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	183 # don't forget to re.escape
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	184 if re.search(re.escape(useragent), agent):
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	185 return 1
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	186 return 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	187
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	188 def allowance(self, filename):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	189 """Preconditions:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	190 - our agent applies to this entry
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	191 - filename is URL decoded"""
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	192 for line in self.rulelines:
16579 a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	193 _debug((filename, str(line), line.allowance))
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	194 if line.applies_to(filename):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	195 return line.allowance
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	196 return 1
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	197
16579 a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	198 class URLopener(urllib.FancyURLopener):
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	199 def __init__(self, *args):
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	200 apply(urllib.FancyURLopener.__init__, (self,) + args)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	201 self.errcode = 200
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	202 self.tries = 0
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	203 self.maxtries = 10
16616 e2a63584566f Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 16579 diff changeset	204
16579 a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	205 def http_error_default(self, url, fp, errcode, errmsg, headers):
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	206 self.errcode = errcode
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	207 return urllib.FancyURLopener.http_error_default(self, url, fp, errcode,
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	208 errmsg, headers)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	209
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	210 def http_error_302(self, url, fp, errcode, errmsg, headers, data=None):
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	211 self.tries += 1
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	212 if self.tries >= self.maxtries:
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	213 return self.http_error_default(url, fp, 500,
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	214 "Internal Server Error: Redirect Recursion",
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	215 headers)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	216 result = urllib.FancyURLopener.http_error_302(self, url, fp, errcode,
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	217 errmsg, headers, data)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	218 self.tries = 0
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	219 return result
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	220
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	221 def _check(a,b):
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	222 if not b:
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	223 ac = "access denied"
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	224 else:
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	225 ac = "access allowed"
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	226 if a!=b:
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	227 print "failed"
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	228 else:
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	229 print "ok (%s)" % ac
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	230 print
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	231
12062 c43b38079228 Skip Montanaro: Guido van Rossum <guido@python.org> parents: 7887 diff changeset	232 def _test():
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	233 global debug
5063 f0731fa4447f Skip Montanaro's robots.txt parser. Guido van Rossum <guido@python.org> parents: diff changeset	234 rp = RobotFileParser()
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	235 debug = 1
16579 a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	236
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	237 # robots.txt that exists, gotten to by redirection
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	238 rp.set_url('http://www.musi-cal.com/robots.txt')
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	239 rp.read()
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	240
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	241 # test for re.escape
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	242 _check(rp.can_fetch('*', 'http://www.musi-cal.com/'), 1)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	243 # this should match the first rule, which is a disallow
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	244 _check(rp.can_fetch('', 'http://www.musi-cal.com/'), 0)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	245 # various cherry pickers
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	246 _check(rp.can_fetch('CherryPickerSE',
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	247 'http://www.musi-cal.com/cgi-bin/event-search'
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	248 '?city=San+Francisco'), 0)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	249 _check(rp.can_fetch('CherryPickerSE/1.0',
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	250 'http://www.musi-cal.com/cgi-bin/event-search'
16579 a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	251 '?city=San+Francisco'), 0)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	252 _check(rp.can_fetch('CherryPickerSE/1.5',
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	253 'http://www.musi-cal.com/cgi-bin/event-search'
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	254 '?city=San+Francisco'), 0)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	255 # case sensitivity
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	256 _check(rp.can_fetch('ExtractorPro', 'http://www.musi-cal.com/blubba'), 0)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	257 _check(rp.can_fetch('extractorpro', 'http://www.musi-cal.com/blubba'), 0)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	258 # substring test
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	259 _check(rp.can_fetch('toolpak/1.1', 'http://www.musi-cal.com/blubba'), 0)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	260 # tests for catch-all * agent
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	261 _check(rp.can_fetch('spam', 'http://www.musi-cal.com/search'), 0)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	262 _check(rp.can_fetch('spam', 'http://www.musi-cal.com/Musician/me'), 1)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	263 _check(rp.can_fetch('spam', 'http://www.musi-cal.com/'), 1)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	264 _check(rp.can_fetch('spam', 'http://www.musi-cal.com/'), 1)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	265
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	266 # robots.txt that does not exist
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	267 rp.set_url('http://www.lycos.com/robots.txt')
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	268 rp.read()
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam Skip Montanaro <skip@pobox.com> parents: 16494 diff changeset	269 _check(rp.can_fetch('Mozilla', 'http://www.lycos.com/search'), 1)
5063 f0731fa4447f Skip Montanaro's robots.txt parser. Guido van Rossum <guido@python.org> parents: diff changeset	270
16067 fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. Skip Montanaro <skip@pobox.com> parents: 12062 diff changeset	271 if __name__ == '__main__':
12062 c43b38079228 Skip Montanaro: Guido van Rossum <guido@python.org> parents: 7887 diff changeset	272 _test()

Mercurial > cpython-lru_cache-weakref

annotate Lib/robotparser.py @ 25838:8540887bcec6 2.2