Mercurial > cpython-lru_cache-weakref

annotate Lib/robotparser.py @ 44825:ef859cd754a7 legacy-trunk

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Fixed a wrong assumption in configure.in and Include/pyport.h. The is finite function is not called isfinite() but finite(). Sorry, my fault. :)
author Christian Heimes <christian@cheimes.de>
date Sun, 20 Jan 2008 14:28:28 +0000
parents e6a9fcd1b44e
children 1ea4d1b18aed
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
1 """ robotparser.py
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
2
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
3 Copyright (C) 2000 Bastian Kleineidam
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
4
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
5 You can choose between two licenses when using this package:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
6 1) GNU GPLv2
22370
c210562e1558 Patch #499513: use readline() instead of readlines(). Removed the
Martin v. Löwis <martin@v.loewis.de>
parents: 22088
diff changeset
7 2) PSF license for Python 2.2
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
8
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
9 The robots.txt Exclusion Protocol is implemented as specified in
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
10 http://info.webcrawler.com/mak/projects/robots/norobots-rfc.html
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
11 """
23591
acbe8e7f99d9 Remove import of re, it is not used
Neal Norwitz <nnorwitz@gmail.com>
parents: 23565
diff changeset
12 import urlparse,urllib
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
13
16070
b5345e99596b added __all__ lists to a number of Python modules
Skip Montanaro <skip@pobox.com>
parents: 16067
diff changeset
14 __all__ = ["RobotFileParser"]
b5345e99596b added __all__ lists to a number of Python modules
Skip Montanaro <skip@pobox.com>
parents: 16067
diff changeset
15
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
16 debug = 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
17
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
18 def _debug(msg):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
19 if debug: print msg
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
20
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
21
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
22 class RobotFileParser:
23565
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
23 """ This class provides a set of methods to read, parse and answer
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
24 questions about a single robots.txt file.
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
25
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
26 """
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
27
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
28 def __init__(self, url=''):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
29 self.entries = []
22088
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
30 self.default_entry = None
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
31 self.disallow_all = False
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
32 self.allow_all = False
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
33 self.set_url(url)
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
34 self.last_checked = 0
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
35
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
36 def mtime(self):
23565
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
37 """Returns the time the robots.txt file was last fetched.
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
38
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
39 This is useful for long-running web spiders that need to
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
40 check for new robots.txt files periodically.
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
41
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
42 """
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
43 return self.last_checked
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
44
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
45 def modified(self):
23565
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
46 """Sets the time the robots.txt file was last fetched to the
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
47 current time.
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
48
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
49 """
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
50 import time
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
51 self.last_checked = time.time()
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
52
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
53 def set_url(self, url):
23565
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
54 """Sets the URL referring to a robots.txt file."""
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
55 self.url = url
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
56 self.host, self.path = urlparse.urlparse(url)[1:3]
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
57
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
58 def read(self):
23565
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
59 """Reads the robots.txt URL and feeds it to the parser."""
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
60 opener = URLopener()
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
61 f = opener.open(self.url)
22370
c210562e1558 Patch #499513: use readline() instead of readlines(). Removed the
Martin v. Löwis <martin@v.loewis.de>
parents: 22088
diff changeset
62 lines = []
c210562e1558 Patch #499513: use readline() instead of readlines(). Removed the
Martin v. Löwis <martin@v.loewis.de>
parents: 22088
diff changeset
63 line = f.readline()
c210562e1558 Patch #499513: use readline() instead of readlines(). Removed the
Martin v. Löwis <martin@v.loewis.de>
parents: 22088
diff changeset
64 while line:
c210562e1558 Patch #499513: use readline() instead of readlines(). Removed the
Martin v. Löwis <martin@v.loewis.de>
parents: 22088
diff changeset
65 lines.append(line.strip())
c210562e1558 Patch #499513: use readline() instead of readlines(). Removed the
Martin v. Löwis <martin@v.loewis.de>
parents: 22088
diff changeset
66 line = f.readline()
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
67 self.errcode = opener.errcode
41146
8fa4e980b589 Patch #1555098: use str.join() instead of repeated string
Georg Brandl <georg@python.org>
parents: 33240
diff changeset
68 if self.errcode in (401, 403):
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
69 self.disallow_all = True
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
70 _debug("disallow all")
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
71 elif self.errcode >= 400:
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
72 self.allow_all = True
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
73 _debug("allow all")
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
74 elif self.errcode == 200 and lines:
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
75 _debug("parse lines")
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
76 self.parse(lines)
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
77
22371
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
78 def _add_entry(self, entry):
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
79 if "*" in entry.useragents:
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
80 # the default entry is considered last
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
81 self.default_entry = entry
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
82 else:
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
83 self.entries.append(entry)
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
84
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
85 def parse(self, lines):
31653
f2600448e85c SF patch #911431: robot.txt must be robots.txt
Raymond Hettinger <python@rcn.com>
parents: 28089
diff changeset
86 """parse the input lines from a robots.txt file.
16074
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
87 We allow that a user-agent: line is not preceded by
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
88 one or more blank lines."""
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
89 state = 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
90 linenumber = 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
91 entry = Entry()
16074
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
92
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
93 for line in lines:
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
94 linenumber = linenumber + 1
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
95 if not line:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
96 if state==1:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
97 _debug("line %d: warning: you should insert"
16074
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
98 " allow: or disallow: directives below any"
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
99 " user-agent: line" % linenumber)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
100 entry = Entry()
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
101 state = 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
102 elif state==2:
22371
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
103 self._add_entry(entry)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
104 entry = Entry()
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
105 state = 0
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
106 # remove optional comment and strip line
16494
f4d15fdd15dd String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents: 16074
diff changeset
107 i = line.find('#')
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
108 if i>=0:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
109 line = line[:i]
16494
f4d15fdd15dd String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents: 16074
diff changeset
110 line = line.strip()
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
111 if not line:
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
112 continue
16494
f4d15fdd15dd String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents: 16074
diff changeset
113 line = line.split(':', 1)
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
114 if len(line) == 2:
16494
f4d15fdd15dd String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents: 16074
diff changeset
115 line[0] = line[0].strip().lower()
22088
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
116 line[1] = urllib.unquote(line[1].strip())
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
117 if line[0] == "user-agent":
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
118 if state==2:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
119 _debug("line %d: warning: you should insert a blank"
16074
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
120 " line before any user-agent"
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
121 " directive" % linenumber)
22371
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
122 self._add_entry(entry)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
123 entry = Entry()
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
124 entry.useragents.append(line[1])
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
125 state = 1
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
126 elif line[0] == "disallow":
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
127 if state==0:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
128 _debug("line %d: error: you must insert a user-agent:"
16074
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
129 " directive before this line" % linenumber)
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
130 else:
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
131 entry.rulelines.append(RuleLine(line[1], False))
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
132 state = 2
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
133 elif line[0] == "allow":
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
134 if state==0:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
135 _debug("line %d: error: you must insert a user-agent:"
16074
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
136 " directive before this line" % linenumber)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
137 else:
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
138 entry.rulelines.append(RuleLine(line[1], True))
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
139 else:
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
140 _debug("line %d: warning: unknown key %s" % (linenumber,
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
141 line[0]))
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
142 else:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
143 _debug("line %d: error: malformed line %s"%(linenumber, line))
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
144 if state==2:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
145 self.entries.append(entry)
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
146 _debug("Parsed rules:\n%s" % str(self))
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
147
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
148
12062
c43b38079228 Skip Montanaro:
Guido van Rossum <guido@python.org>
parents: 7887
diff changeset
149 def can_fetch(self, useragent, url):
c43b38079228 Skip Montanaro:
Guido van Rossum <guido@python.org>
parents: 7887
diff changeset
150 """using the parsed robots.txt decide if useragent can fetch url"""
31653
f2600448e85c SF patch #911431: robot.txt must be robots.txt
Raymond Hettinger <python@rcn.com>
parents: 28089
diff changeset
151 _debug("Checking robots.txt allowance for:\n user agent: %s\n url: %s" %
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
152 (useragent, url))
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
153 if self.disallow_all:
22830
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
154 return False
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
155 if self.allow_all:
22830
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
156 return True
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
157 # search for given user agent matches
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
158 # the first match counts
22088
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
159 url = urllib.quote(urlparse.urlparse(urllib.unquote(url))[2]) or "/"
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
160 for entry in self.entries:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
161 if entry.applies_to(useragent):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
162 return entry.allowance(url)
22088
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
163 # try the default entry last
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
164 if self.default_entry:
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
165 return self.default_entry.allowance(url)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
166 # agent not found ==> access granted
22830
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
167 return True
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
168
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
169
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
170 def __str__(self):
41146
8fa4e980b589 Patch #1555098: use str.join() instead of repeated string
Georg Brandl <georg@python.org>
parents: 33240
diff changeset
171 return ''.join([str(entry) + "\n" for entry in self.entries])
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
172
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
173
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
174 class RuleLine:
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
175 """A rule line is a single "Allow:" (allowance==True) or "Disallow:"
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
176 (allowance==False) followed by a path."""
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
177 def __init__(self, path, allowance):
22088
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
178 if path == '' and not allowance:
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
179 # an empty value means allow all
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
180 allowance = True
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
181 self.path = urllib.quote(path)
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
182 self.allowance = allowance
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
183
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
184 def applies_to(self, filename):
22088
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
185 return self.path=="*" or filename.startswith(self.path)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
186
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
187 def __str__(self):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
188 return (self.allowance and "Allow" or "Disallow")+": "+self.path
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
189
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
190
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
191 class Entry:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
192 """An entry has one or more user-agents and zero or more rulelines"""
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
193 def __init__(self):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
194 self.useragents = []
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
195 self.rulelines = []
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
196
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
197 def __str__(self):
41146
8fa4e980b589 Patch #1555098: use str.join() instead of repeated string
Georg Brandl <georg@python.org>
parents: 33240
diff changeset
198 ret = []
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
199 for agent in self.useragents:
41146
8fa4e980b589 Patch #1555098: use str.join() instead of repeated string
Georg Brandl <georg@python.org>
parents: 33240
diff changeset
200 ret.extend(["User-agent: ", agent, "\n"])
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
201 for line in self.rulelines:
41146
8fa4e980b589 Patch #1555098: use str.join() instead of repeated string
Georg Brandl <georg@python.org>
parents: 33240
diff changeset
202 ret.extend([str(line), "\n"])
8fa4e980b589 Patch #1555098: use str.join() instead of repeated string
Georg Brandl <georg@python.org>
parents: 33240
diff changeset
203 return ''.join(ret)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
204
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
205 def applies_to(self, useragent):
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
206 """check if this entry applies to the specified agent"""
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
207 # split the name token and make it lower case
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
208 useragent = useragent.split("/")[0].lower()
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
209 for agent in self.useragents:
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
210 if agent=='*':
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
211 # we have the catch-all agent
22830
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
212 return True
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
213 agent = agent.lower()
31967
3d046d063a84 Replace str.find()!=1 with the more readable "in" operator.
Raymond Hettinger <python@rcn.com>
parents: 31653
diff changeset
214 if agent in useragent:
22830
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
215 return True
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
216 return False
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
217
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
218 def allowance(self, filename):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
219 """Preconditions:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
220 - our agent applies to this entry
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
221 - filename is URL decoded"""
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
222 for line in self.rulelines:
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
223 _debug((filename, str(line), line.allowance))
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
224 if line.applies_to(filename):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
225 return line.allowance
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
226 return True
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
227
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
228 class URLopener(urllib.FancyURLopener):
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
229 def __init__(self, *args):
28089
14f65d661084 Get rid of many apply() calls.
Guido van Rossum <guido@python.org>
parents: 23591
diff changeset
230 urllib.FancyURLopener.__init__(self, *args)
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
231 self.errcode = 200
16616
e2a63584566f Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16579
diff changeset
232
42956
e6a9fcd1b44e fixes 813986
Skip Montanaro <skip@pobox.com>
parents: 41146
diff changeset
233 def prompt_user_passwd(self, host, realm):
e6a9fcd1b44e fixes 813986
Skip Montanaro <skip@pobox.com>
parents: 41146
diff changeset
234 ## If robots.txt file is accessible only with a password,
e6a9fcd1b44e fixes 813986
Skip Montanaro <skip@pobox.com>
parents: 41146
diff changeset
235 ## we act as if the file wasn't there.
e6a9fcd1b44e fixes 813986
Skip Montanaro <skip@pobox.com>
parents: 41146
diff changeset
236 return None, None
e6a9fcd1b44e fixes 813986
Skip Montanaro <skip@pobox.com>
parents: 41146
diff changeset
237
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
238 def http_error_default(self, url, fp, errcode, errmsg, headers):
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
239 self.errcode = errcode
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
240 return urllib.FancyURLopener.http_error_default(self, url, fp, errcode,
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
241 errmsg, headers)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
242
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
243 def _check(a,b):
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
244 if not b:
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
245 ac = "access denied"
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
246 else:
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
247 ac = "access allowed"
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
248 if a!=b:
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
249 print "failed"
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
250 else:
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
251 print "ok (%s)" % ac
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
252 print
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
253
12062
c43b38079228 Skip Montanaro:
Guido van Rossum <guido@python.org>
parents: 7887
diff changeset
254 def _test():
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
255 global debug
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
256 rp = RobotFileParser()
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
257 debug = 1
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
258
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
259 # robots.txt that exists, gotten to by redirection
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
260 rp.set_url('http://www.musi-cal.com/robots.txt')
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
261 rp.read()
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
262
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
263 # test for re.escape
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
264 _check(rp.can_fetch('*', 'http://www.musi-cal.com/'), 1)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
265 # this should match the first rule, which is a disallow
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
266 _check(rp.can_fetch('', 'http://www.musi-cal.com/'), 0)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
267 # various cherry pickers
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
268 _check(rp.can_fetch('CherryPickerSE',
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
269 'http://www.musi-cal.com/cgi-bin/event-search'
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
270 '?city=San+Francisco'), 0)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
271 _check(rp.can_fetch('CherryPickerSE/1.0',
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
272 'http://www.musi-cal.com/cgi-bin/event-search'
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
273 '?city=San+Francisco'), 0)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
274 _check(rp.can_fetch('CherryPickerSE/1.5',
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
275 'http://www.musi-cal.com/cgi-bin/event-search'
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
276 '?city=San+Francisco'), 0)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
277 # case sensitivity
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
278 _check(rp.can_fetch('ExtractorPro', 'http://www.musi-cal.com/blubba'), 0)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
279 _check(rp.can_fetch('extractorpro', 'http://www.musi-cal.com/blubba'), 0)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
280 # substring test
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
281 _check(rp.can_fetch('toolpak/1.1', 'http://www.musi-cal.com/blubba'), 0)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
282 # tests for catch-all * agent
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
283 _check(rp.can_fetch('spam', 'http://www.musi-cal.com/search'), 0)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
284 _check(rp.can_fetch('spam', 'http://www.musi-cal.com/Musician/me'), 1)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
285 _check(rp.can_fetch('spam', 'http://www.musi-cal.com/'), 1)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
286 _check(rp.can_fetch('spam', 'http://www.musi-cal.com/'), 1)
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
287
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
288 # robots.txt that does not exist
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
289 rp.set_url('http://www.lycos.com/robots.txt')
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
290 rp.read()
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
291 _check(rp.can_fetch('Mozilla', 'http://www.lycos.com/search'), 1)
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
292
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
293 if __name__ == '__main__':
12062
c43b38079228 Skip Montanaro:
Guido van Rossum <guido@python.org>
parents: 7887
diff changeset
294 _test()