Mercurial > cpython-lru_cache-weakref

annotate Lib/robotparser.py @ 58765:2ba7677bfa95 legacy-trunk

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
testCopysign was defined twice in test_math; combine the definitions
author Mark Dickinson <dickinsm@gmail.com>
date Sat, 06 Feb 2010 23:11:25 +0000
parents 9f9e5fe63ec2
children 2c253f6e02e7 0bb47c51e933
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
1 """ robotparser.py
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
2
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
3 Copyright (C) 2000 Bastian Kleineidam
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
4
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
5 You can choose between two licenses when using this package:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
6 1) GNU GPLv2
22370
c210562e1558 Patch #499513: use readline() instead of readlines(). Removed the
Martin v. Löwis <martin@v.loewis.de>
parents: 22088
diff changeset
7 2) PSF license for Python 2.2
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
8
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
9 The robots.txt Exclusion Protocol is implemented as specified in
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
10 http://info.webcrawler.com/mak/projects/robots/norobots-rfc.html
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
11 """
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
12 import urlparse
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
13 import urllib
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
14
16070
b5345e99596b added __all__ lists to a number of Python modules
Skip Montanaro <skip@pobox.com>
parents: 16067
diff changeset
15 __all__ = ["RobotFileParser"]
b5345e99596b added __all__ lists to a number of Python modules
Skip Montanaro <skip@pobox.com>
parents: 16067
diff changeset
16
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
17
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
18 class RobotFileParser:
23565
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
19 """ This class provides a set of methods to read, parse and answer
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
20 questions about a single robots.txt file.
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
21
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
22 """
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
23
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
24 def __init__(self, url=''):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
25 self.entries = []
22088
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
26 self.default_entry = None
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
27 self.disallow_all = False
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
28 self.allow_all = False
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
29 self.set_url(url)
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
30 self.last_checked = 0
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
31
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
32 def mtime(self):
23565
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
33 """Returns the time the robots.txt file was last fetched.
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
34
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
35 This is useful for long-running web spiders that need to
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
36 check for new robots.txt files periodically.
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
37
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
38 """
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
39 return self.last_checked
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
40
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
41 def modified(self):
23565
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
42 """Sets the time the robots.txt file was last fetched to the
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
43 current time.
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
44
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
45 """
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
46 import time
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
47 self.last_checked = time.time()
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
48
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
49 def set_url(self, url):
23565
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
50 """Sets the URL referring to a robots.txt file."""
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
51 self.url = url
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
52 self.host, self.path = urlparse.urlparse(url)[1:3]
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
53
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
54 def read(self):
23565
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
55 """Reads the robots.txt URL and feeds it to the parser."""
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
56 opener = URLopener()
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
57 f = opener.open(self.url)
47939
2c9d0a7e3bc4 #1778443 robotparser fixes from Aristotelis Mikropoulos
Benjamin Peterson <benjamin@python.org>
parents: 46591
diff changeset
58 lines = [line.strip() for line in f]
2c9d0a7e3bc4 #1778443 robotparser fixes from Aristotelis Mikropoulos
Benjamin Peterson <benjamin@python.org>
parents: 46591
diff changeset
59 f.close()
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
60 self.errcode = opener.errcode
41146
8fa4e980b589 Patch #1555098: use str.join() instead of repeated string
Georg Brandl <georg@python.org>
parents: 33240
diff changeset
61 if self.errcode in (401, 403):
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
62 self.disallow_all = True
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
63 elif self.errcode >= 400:
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
64 self.allow_all = True
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
65 elif self.errcode == 200 and lines:
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
66 self.parse(lines)
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
67
22371
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
68 def _add_entry(self, entry):
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
69 if "*" in entry.useragents:
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
70 # the default entry is considered last
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
71 self.default_entry = entry
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
72 else:
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
73 self.entries.append(entry)
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
74
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
75 def parse(self, lines):
31653
f2600448e85c SF patch #911431: robot.txt must be robots.txt
Raymond Hettinger <python@rcn.com>
parents: 28089
diff changeset
76 """parse the input lines from a robots.txt file.
16074
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
77 We allow that a user-agent: line is not preceded by
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
78 one or more blank lines."""
48085
9f9e5fe63ec2 Close issue 3437 - missing state change when Allow lines are processed.
Skip Montanaro <skip@pobox.com>
parents: 47939
diff changeset
79 # states:
9f9e5fe63ec2 Close issue 3437 - missing state change when Allow lines are processed.
Skip Montanaro <skip@pobox.com>
parents: 47939
diff changeset
80 # 0: start state
9f9e5fe63ec2 Close issue 3437 - missing state change when Allow lines are processed.
Skip Montanaro <skip@pobox.com>
parents: 47939
diff changeset
81 # 1: saw user-agent line
9f9e5fe63ec2 Close issue 3437 - missing state change when Allow lines are processed.
Skip Montanaro <skip@pobox.com>
parents: 47939
diff changeset
82 # 2: saw an allow or disallow line
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
83 state = 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
84 linenumber = 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
85 entry = Entry()
16074
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
86
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
87 for line in lines:
47939
2c9d0a7e3bc4 #1778443 robotparser fixes from Aristotelis Mikropoulos
Benjamin Peterson <benjamin@python.org>
parents: 46591
diff changeset
88 linenumber += 1
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
89 if not line:
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
90 if state == 1:
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
91 entry = Entry()
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
92 state = 0
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
93 elif state == 2:
22371
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
94 self._add_entry(entry)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
95 entry = Entry()
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
96 state = 0
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
97 # remove optional comment and strip line
16494
f4d15fdd15dd String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents: 16074
diff changeset
98 i = line.find('#')
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
99 if i >= 0:
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
100 line = line[:i]
16494
f4d15fdd15dd String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents: 16074
diff changeset
101 line = line.strip()
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
102 if not line:
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
103 continue
16494
f4d15fdd15dd String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents: 16074
diff changeset
104 line = line.split(':', 1)
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
105 if len(line) == 2:
16494
f4d15fdd15dd String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents: 16074
diff changeset
106 line[0] = line[0].strip().lower()
22088
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
107 line[1] = urllib.unquote(line[1].strip())
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
108 if line[0] == "user-agent":
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
109 if state == 2:
22371
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
110 self._add_entry(entry)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
111 entry = Entry()
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
112 entry.useragents.append(line[1])
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
113 state = 1
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
114 elif line[0] == "disallow":
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
115 if state != 0:
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
116 entry.rulelines.append(RuleLine(line[1], False))
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
117 state = 2
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
118 elif line[0] == "allow":
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
119 if state != 0:
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
120 entry.rulelines.append(RuleLine(line[1], True))
48085
9f9e5fe63ec2 Close issue 3437 - missing state change when Allow lines are processed.
Skip Montanaro <skip@pobox.com>
parents: 47939
diff changeset
121 state = 2
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
122 if state == 2:
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
123 self.entries.append(entry)
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
124
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
125
12062
c43b38079228 Skip Montanaro:
Guido van Rossum <guido@python.org>
parents: 7887
diff changeset
126 def can_fetch(self, useragent, url):
c43b38079228 Skip Montanaro:
Guido van Rossum <guido@python.org>
parents: 7887
diff changeset
127 """using the parsed robots.txt decide if useragent can fetch url"""
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
128 if self.disallow_all:
22830
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
129 return False
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
130 if self.allow_all:
22830
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
131 return True
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
132 # search for given user agent matches
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
133 # the first match counts
22088
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
134 url = urllib.quote(urlparse.urlparse(urllib.unquote(url))[2]) or "/"
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
135 for entry in self.entries:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
136 if entry.applies_to(useragent):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
137 return entry.allowance(url)
22088
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
138 # try the default entry last
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
139 if self.default_entry:
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
140 return self.default_entry.allowance(url)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
141 # agent not found ==> access granted
22830
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
142 return True
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
143
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
144
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
145 def __str__(self):
41146
8fa4e980b589 Patch #1555098: use str.join() instead of repeated string
Georg Brandl <georg@python.org>
parents: 33240
diff changeset
146 return ''.join([str(entry) + "\n" for entry in self.entries])
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
147
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
148
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
149 class RuleLine:
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
150 """A rule line is a single "Allow:" (allowance==True) or "Disallow:"
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
151 (allowance==False) followed by a path."""
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
152 def __init__(self, path, allowance):
22088
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
153 if path == '' and not allowance:
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
154 # an empty value means allow all
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
155 allowance = True
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
156 self.path = urllib.quote(path)
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
157 self.allowance = allowance
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
158
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
159 def applies_to(self, filename):
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
160 return self.path == "*" or filename.startswith(self.path)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
161
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
162 def __str__(self):
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
163 return (self.allowance and "Allow" or "Disallow") + ": " + self.path
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
164
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
165
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
166 class Entry:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
167 """An entry has one or more user-agents and zero or more rulelines"""
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
168 def __init__(self):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
169 self.useragents = []
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
170 self.rulelines = []
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
171
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
172 def __str__(self):
41146
8fa4e980b589 Patch #1555098: use str.join() instead of repeated string
Georg Brandl <georg@python.org>
parents: 33240
diff changeset
173 ret = []
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
174 for agent in self.useragents:
41146
8fa4e980b589 Patch #1555098: use str.join() instead of repeated string
Georg Brandl <georg@python.org>
parents: 33240
diff changeset
175 ret.extend(["User-agent: ", agent, "\n"])
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
176 for line in self.rulelines:
41146
8fa4e980b589 Patch #1555098: use str.join() instead of repeated string
Georg Brandl <georg@python.org>
parents: 33240
diff changeset
177 ret.extend([str(line), "\n"])
8fa4e980b589 Patch #1555098: use str.join() instead of repeated string
Georg Brandl <georg@python.org>
parents: 33240
diff changeset
178 return ''.join(ret)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
179
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
180 def applies_to(self, useragent):
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
181 """check if this entry applies to the specified agent"""
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
182 # split the name token and make it lower case
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
183 useragent = useragent.split("/")[0].lower()
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
184 for agent in self.useragents:
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
185 if agent == '*':
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
186 # we have the catch-all agent
22830
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
187 return True
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
188 agent = agent.lower()
31967
3d046d063a84 Replace str.find()!=1 with the more readable "in" operator.
Raymond Hettinger <python@rcn.com>
parents: 31653
diff changeset
189 if agent in useragent:
22830
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
190 return True
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
191 return False
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
192
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
193 def allowance(self, filename):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
194 """Preconditions:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
195 - our agent applies to this entry
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
196 - filename is URL decoded"""
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
197 for line in self.rulelines:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
198 if line.applies_to(filename):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
199 return line.allowance
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
200 return True
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
201
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
202 class URLopener(urllib.FancyURLopener):
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
203 def __init__(self, *args):
28089
14f65d661084 Get rid of many apply() calls.
Guido van Rossum <guido@python.org>
parents: 23591
diff changeset
204 urllib.FancyURLopener.__init__(self, *args)
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
205 self.errcode = 200
16616
e2a63584566f Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16579
diff changeset
206
42956
e6a9fcd1b44e fixes 813986
Skip Montanaro <skip@pobox.com>
parents: 41146
diff changeset
207 def prompt_user_passwd(self, host, realm):
e6a9fcd1b44e fixes 813986
Skip Montanaro <skip@pobox.com>
parents: 41146
diff changeset
208 ## If robots.txt file is accessible only with a password,
e6a9fcd1b44e fixes 813986
Skip Montanaro <skip@pobox.com>
parents: 41146
diff changeset
209 ## we act as if the file wasn't there.
e6a9fcd1b44e fixes 813986
Skip Montanaro <skip@pobox.com>
parents: 41146
diff changeset
210 return None, None
e6a9fcd1b44e fixes 813986
Skip Montanaro <skip@pobox.com>
parents: 41146
diff changeset
211
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
212 def http_error_default(self, url, fp, errcode, errmsg, headers):
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
213 self.errcode = errcode
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
214 return urllib.FancyURLopener.http_error_default(self, url, fp, errcode,
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
215 errmsg, headers)