Mercurial > cpython-lru_cache-weakref

annotate Lib/robotparser.py @ 47787:b32101c19bf8 legacy-trunk

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Grammar fixes
author Andrew M. Kuchling <amk@amk.ca>
date Fri, 20 Jun 2008 23:14:32 +0000
parents 1ea4d1b18aed
children 2c9d0a7e3bc4
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
1 """ robotparser.py
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
2
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
3 Copyright (C) 2000 Bastian Kleineidam
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
4
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
5 You can choose between two licenses when using this package:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
6 1) GNU GPLv2
22370
c210562e1558 Patch #499513: use readline() instead of readlines(). Removed the
Martin v. Löwis <martin@v.loewis.de>
parents: 22088
diff changeset
7 2) PSF license for Python 2.2
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
8
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
9 The robots.txt Exclusion Protocol is implemented as specified in
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
10 http://info.webcrawler.com/mak/projects/robots/norobots-rfc.html
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
11 """
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
12 import urlparse
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
13 import urllib
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
14
16070
b5345e99596b added __all__ lists to a number of Python modules
Skip Montanaro <skip@pobox.com>
parents: 16067
diff changeset
15 __all__ = ["RobotFileParser"]
b5345e99596b added __all__ lists to a number of Python modules
Skip Montanaro <skip@pobox.com>
parents: 16067
diff changeset
16
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
17
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
18 class RobotFileParser:
23565
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
19 """ This class provides a set of methods to read, parse and answer
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
20 questions about a single robots.txt file.
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
21
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
22 """
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
23
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
24 def __init__(self, url=''):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
25 self.entries = []
22088
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
26 self.default_entry = None
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
27 self.disallow_all = False
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
28 self.allow_all = False
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
29 self.set_url(url)
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
30 self.last_checked = 0
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
31
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
32 def mtime(self):
23565
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
33 """Returns the time the robots.txt file was last fetched.
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
34
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
35 This is useful for long-running web spiders that need to
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
36 check for new robots.txt files periodically.
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
37
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
38 """
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
39 return self.last_checked
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
40
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
41 def modified(self):
23565
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
42 """Sets the time the robots.txt file was last fetched to the
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
43 current time.
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
44
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
45 """
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
46 import time
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
47 self.last_checked = time.time()
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
48
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
49 def set_url(self, url):
23565
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
50 """Sets the URL referring to a robots.txt file."""
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
51 self.url = url
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
52 self.host, self.path = urlparse.urlparse(url)[1:3]
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
53
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
54 def read(self):
23565
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
55 """Reads the robots.txt URL and feeds it to the parser."""
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
56 opener = URLopener()
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
57 f = opener.open(self.url)
22370
c210562e1558 Patch #499513: use readline() instead of readlines(). Removed the
Martin v. Löwis <martin@v.loewis.de>
parents: 22088
diff changeset
58 lines = []
c210562e1558 Patch #499513: use readline() instead of readlines(). Removed the
Martin v. Löwis <martin@v.loewis.de>
parents: 22088
diff changeset
59 line = f.readline()
c210562e1558 Patch #499513: use readline() instead of readlines(). Removed the
Martin v. Löwis <martin@v.loewis.de>
parents: 22088
diff changeset
60 while line:
c210562e1558 Patch #499513: use readline() instead of readlines(). Removed the
Martin v. Löwis <martin@v.loewis.de>
parents: 22088
diff changeset
61 lines.append(line.strip())
c210562e1558 Patch #499513: use readline() instead of readlines(). Removed the
Martin v. Löwis <martin@v.loewis.de>
parents: 22088
diff changeset
62 line = f.readline()
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
63 self.errcode = opener.errcode
41146
8fa4e980b589 Patch #1555098: use str.join() instead of repeated string
Georg Brandl <georg@python.org>
parents: 33240
diff changeset
64 if self.errcode in (401, 403):
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
65 self.disallow_all = True
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
66 elif self.errcode >= 400:
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
67 self.allow_all = True
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
68 elif self.errcode == 200 and lines:
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
69 self.parse(lines)
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
70
22371
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
71 def _add_entry(self, entry):
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
72 if "*" in entry.useragents:
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
73 # the default entry is considered last
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
74 self.default_entry = entry
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
75 else:
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
76 self.entries.append(entry)
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
77
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
78 def parse(self, lines):
31653
f2600448e85c SF patch #911431: robot.txt must be robots.txt
Raymond Hettinger <python@rcn.com>
parents: 28089
diff changeset
79 """parse the input lines from a robots.txt file.
16074
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
80 We allow that a user-agent: line is not preceded by
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
81 one or more blank lines."""
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
82 state = 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
83 linenumber = 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
84 entry = Entry()
16074
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
85
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
86 for line in lines:
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
87 linenumber = linenumber + 1
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
88 if not line:
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
89 if state == 1:
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
90 entry = Entry()
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
91 state = 0
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
92 elif state == 2:
22371
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
93 self._add_entry(entry)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
94 entry = Entry()
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
95 state = 0
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
96 # remove optional comment and strip line
16494
f4d15fdd15dd String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents: 16074
diff changeset
97 i = line.find('#')
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
98 if i >= 0:
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
99 line = line[:i]
16494
f4d15fdd15dd String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents: 16074
diff changeset
100 line = line.strip()
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
101 if not line:
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
102 continue
16494
f4d15fdd15dd String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents: 16074
diff changeset
103 line = line.split(':', 1)
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
104 if len(line) == 2:
16494
f4d15fdd15dd String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents: 16074
diff changeset
105 line[0] = line[0].strip().lower()
22088
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
106 line[1] = urllib.unquote(line[1].strip())
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
107 if line[0] == "user-agent":
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
108 if state == 2:
22371
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
109 self._add_entry(entry)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
110 entry = Entry()
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
111 entry.useragents.append(line[1])
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
112 state = 1
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
113 elif line[0] == "disallow":
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
114 if state != 0:
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
115 entry.rulelines.append(RuleLine(line[1], False))
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
116 state = 2
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
117 elif line[0] == "allow":
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
118 if state != 0:
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
119 entry.rulelines.append(RuleLine(line[1], True))
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
120 if state == 2:
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
121 self.entries.append(entry)
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
122
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
123
12062
c43b38079228 Skip Montanaro:
Guido van Rossum <guido@python.org>
parents: 7887
diff changeset
124 def can_fetch(self, useragent, url):
c43b38079228 Skip Montanaro:
Guido van Rossum <guido@python.org>
parents: 7887
diff changeset
125 """using the parsed robots.txt decide if useragent can fetch url"""
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
126 if self.disallow_all:
22830
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
127 return False
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
128 if self.allow_all:
22830
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
129 return True
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
130 # search for given user agent matches
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
131 # the first match counts
22088
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
132 url = urllib.quote(urlparse.urlparse(urllib.unquote(url))[2]) or "/"
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
133 for entry in self.entries:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
134 if entry.applies_to(useragent):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
135 return entry.allowance(url)
22088
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
136 # try the default entry last
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
137 if self.default_entry:
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
138 return self.default_entry.allowance(url)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
139 # agent not found ==> access granted
22830
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
140 return True
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
141
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
142
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
143 def __str__(self):
41146
8fa4e980b589 Patch #1555098: use str.join() instead of repeated string
Georg Brandl <georg@python.org>
parents: 33240
diff changeset
144 return ''.join([str(entry) + "\n" for entry in self.entries])
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
145
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
146
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
147 class RuleLine:
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
148 """A rule line is a single "Allow:" (allowance==True) or "Disallow:"
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
149 (allowance==False) followed by a path."""
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
150 def __init__(self, path, allowance):
22088
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
151 if path == '' and not allowance:
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
152 # an empty value means allow all
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
153 allowance = True
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
154 self.path = urllib.quote(path)
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
155 self.allowance = allowance
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
156
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
157 def applies_to(self, filename):
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
158 return self.path == "*" or filename.startswith(self.path)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
159
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
160 def __str__(self):
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
161 return (self.allowance and "Allow" or "Disallow") + ": " + self.path
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
162
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
163
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
164 class Entry:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
165 """An entry has one or more user-agents and zero or more rulelines"""
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
166 def __init__(self):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
167 self.useragents = []
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
168 self.rulelines = []
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
169
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
170 def __str__(self):
41146
8fa4e980b589 Patch #1555098: use str.join() instead of repeated string
Georg Brandl <georg@python.org>
parents: 33240
diff changeset
171 ret = []
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
172 for agent in self.useragents:
41146
8fa4e980b589 Patch #1555098: use str.join() instead of repeated string
Georg Brandl <georg@python.org>
parents: 33240
diff changeset
173 ret.extend(["User-agent: ", agent, "\n"])
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
174 for line in self.rulelines:
41146
8fa4e980b589 Patch #1555098: use str.join() instead of repeated string
Georg Brandl <georg@python.org>
parents: 33240
diff changeset
175 ret.extend([str(line), "\n"])
8fa4e980b589 Patch #1555098: use str.join() instead of repeated string
Georg Brandl <georg@python.org>
parents: 33240
diff changeset
176 return ''.join(ret)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
177
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
178 def applies_to(self, useragent):
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
179 """check if this entry applies to the specified agent"""
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
180 # split the name token and make it lower case
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
181 useragent = useragent.split("/")[0].lower()
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
182 for agent in self.useragents:
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
183 if agent == '*':
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
184 # we have the catch-all agent
22830
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
185 return True
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
186 agent = agent.lower()
31967
3d046d063a84 Replace str.find()!=1 with the more readable "in" operator.
Raymond Hettinger <python@rcn.com>
parents: 31653
diff changeset
187 if agent in useragent:
22830
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
188 return True
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
189 return False
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
190
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
191 def allowance(self, filename):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
192 """Preconditions:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
193 - our agent applies to this entry
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
194 - filename is URL decoded"""
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
195 for line in self.rulelines:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
196 if line.applies_to(filename):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
197 return line.allowance
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
198 return True
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
199
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
200 class URLopener(urllib.FancyURLopener):
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
201 def __init__(self, *args):
28089
14f65d661084 Get rid of many apply() calls.
Guido van Rossum <guido@python.org>
parents: 23591
diff changeset
202 urllib.FancyURLopener.__init__(self, *args)
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
203 self.errcode = 200
16616
e2a63584566f Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16579
diff changeset
204
42956
e6a9fcd1b44e fixes 813986
Skip Montanaro <skip@pobox.com>
parents: 41146
diff changeset
205 def prompt_user_passwd(self, host, realm):
e6a9fcd1b44e fixes 813986
Skip Montanaro <skip@pobox.com>
parents: 41146
diff changeset
206 ## If robots.txt file is accessible only with a password,
e6a9fcd1b44e fixes 813986
Skip Montanaro <skip@pobox.com>
parents: 41146
diff changeset
207 ## we act as if the file wasn't there.
e6a9fcd1b44e fixes 813986
Skip Montanaro <skip@pobox.com>
parents: 41146
diff changeset
208 return None, None
e6a9fcd1b44e fixes 813986
Skip Montanaro <skip@pobox.com>
parents: 41146
diff changeset
209
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
210 def http_error_default(self, url, fp, errcode, errmsg, headers):
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
211 self.errcode = errcode
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
212 return urllib.FancyURLopener.http_error_default(self, url, fp, errcode,
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
213 errmsg, headers)