Mercurial > cpython-lru_cache-weakref

annotate Lib/robotparser.py @ 80661:94de77bd0b4b 2.7

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Issue #16194: document imp.load_dynamic problems when read several extension modules from the same binary object file.
author Andrew Svetlov <andrew.svetlov@gmail.com>
date Thu, 29 Nov 2012 19:51:28 +0200
parents b36cb4602e21
children bcbad715c2ce
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
1 """ robotparser.py
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
2
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
3 Copyright (C) 2000 Bastian Kleineidam
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
4
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
5 You can choose between two licenses when using this package:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
6 1) GNU GPLv2
22370
c210562e1558 Patch #499513: use readline() instead of readlines(). Removed the
Martin v. Löwis <martin@v.loewis.de>
parents: 22088
diff changeset
7 2) PSF license for Python 2.2
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
8
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
9 The robots.txt Exclusion Protocol is implemented as specified in
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
10 http://info.webcrawler.com/mak/projects/robots/norobots-rfc.html
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
11 """
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
12 import urlparse
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
13 import urllib
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
14
16070
b5345e99596b added __all__ lists to a number of Python modules
Skip Montanaro <skip@pobox.com>
parents: 16067
diff changeset
15 __all__ = ["RobotFileParser"]
b5345e99596b added __all__ lists to a number of Python modules
Skip Montanaro <skip@pobox.com>
parents: 16067
diff changeset
16
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
17
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
18 class RobotFileParser:
23565
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
19 """ This class provides a set of methods to read, parse and answer
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
20 questions about a single robots.txt file.
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
21
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
22 """
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
23
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
24 def __init__(self, url=''):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
25 self.entries = []
22088
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
26 self.default_entry = None
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
27 self.disallow_all = False
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
28 self.allow_all = False
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
29 self.set_url(url)
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
30 self.last_checked = 0
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
31
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
32 def mtime(self):
23565
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
33 """Returns the time the robots.txt file was last fetched.
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
34
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
35 This is useful for long-running web spiders that need to
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
36 check for new robots.txt files periodically.
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
37
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
38 """
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
39 return self.last_checked
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
40
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
41 def modified(self):
23565
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
42 """Sets the time the robots.txt file was last fetched to the
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
43 current time.
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
44
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
45 """
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
46 import time
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
47 self.last_checked = time.time()
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
48
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
49 def set_url(self, url):
23565
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
50 """Sets the URL referring to a robots.txt file."""
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
51 self.url = url
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
52 self.host, self.path = urlparse.urlparse(url)[1:3]
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
53
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
54 def read(self):
23565
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
55 """Reads the robots.txt URL and feeds it to the parser."""
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
56 opener = URLopener()
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
57 f = opener.open(self.url)
47939
2c9d0a7e3bc4 #1778443 robotparser fixes from Aristotelis Mikropoulos
Benjamin Peterson <benjamin@python.org>
parents: 46591
diff changeset
58 lines = [line.strip() for line in f]
2c9d0a7e3bc4 #1778443 robotparser fixes from Aristotelis Mikropoulos
Benjamin Peterson <benjamin@python.org>
parents: 46591
diff changeset
59 f.close()
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
60 self.errcode = opener.errcode
41146
8fa4e980b589 Patch #1555098: use str.join() instead of repeated string
Georg Brandl <georg@python.org>
parents: 33240
diff changeset
61 if self.errcode in (401, 403):
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
62 self.disallow_all = True
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
63 elif self.errcode >= 400:
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
64 self.allow_all = True
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
65 elif self.errcode == 200 and lines:
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
66 self.parse(lines)
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
67
22371
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
68 def _add_entry(self, entry):
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
69 if "*" in entry.useragents:
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
70 # the default entry is considered last
63506
2c253f6e02e7 Merged revisions 83449 via svnmerge from
Georg Brandl <georg@python.org>
parents: 48085
diff changeset
71 if self.default_entry is None:
2c253f6e02e7 Merged revisions 83449 via svnmerge from
Georg Brandl <georg@python.org>
parents: 48085
diff changeset
72 # the first default entry wins
2c253f6e02e7 Merged revisions 83449 via svnmerge from
Georg Brandl <georg@python.org>
parents: 48085
diff changeset
73 self.default_entry = entry
22371
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
74 else:
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
75 self.entries.append(entry)
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
76
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
77 def parse(self, lines):
31653
f2600448e85c SF patch #911431: robot.txt must be robots.txt
Raymond Hettinger <python@rcn.com>
parents: 28089
diff changeset
78 """parse the input lines from a robots.txt file.
16074
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
79 We allow that a user-agent: line is not preceded by
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
80 one or more blank lines."""
48085
9f9e5fe63ec2 Close issue 3437 - missing state change when Allow lines are processed.
Skip Montanaro <skip@pobox.com>
parents: 47939
diff changeset
81 # states:
9f9e5fe63ec2 Close issue 3437 - missing state change when Allow lines are processed.
Skip Montanaro <skip@pobox.com>
parents: 47939
diff changeset
82 # 0: start state
9f9e5fe63ec2 Close issue 3437 - missing state change when Allow lines are processed.
Skip Montanaro <skip@pobox.com>
parents: 47939
diff changeset
83 # 1: saw user-agent line
9f9e5fe63ec2 Close issue 3437 - missing state change when Allow lines are processed.
Skip Montanaro <skip@pobox.com>
parents: 47939
diff changeset
84 # 2: saw an allow or disallow line
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
85 state = 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
86 linenumber = 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
87 entry = Entry()
16074
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
88
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
89 for line in lines:
47939
2c9d0a7e3bc4 #1778443 robotparser fixes from Aristotelis Mikropoulos
Benjamin Peterson <benjamin@python.org>
parents: 46591
diff changeset
90 linenumber += 1
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
91 if not line:
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
92 if state == 1:
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
93 entry = Entry()
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
94 state = 0
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
95 elif state == 2:
22371
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
96 self._add_entry(entry)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
97 entry = Entry()
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
98 state = 0
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
99 # remove optional comment and strip line
16494
f4d15fdd15dd String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents: 16074
diff changeset
100 i = line.find('#')
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
101 if i >= 0:
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
102 line = line[:i]
16494
f4d15fdd15dd String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents: 16074
diff changeset
103 line = line.strip()
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
104 if not line:
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
105 continue
16494
f4d15fdd15dd String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents: 16074
diff changeset
106 line = line.split(':', 1)
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
107 if len(line) == 2:
16494
f4d15fdd15dd String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents: 16074
diff changeset
108 line[0] = line[0].strip().lower()
22088
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
109 line[1] = urllib.unquote(line[1].strip())
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
110 if line[0] == "user-agent":
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
111 if state == 2:
22371
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
112 self._add_entry(entry)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
113 entry = Entry()
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
114 entry.useragents.append(line[1])
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
115 state = 1
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
116 elif line[0] == "disallow":
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
117 if state != 0:
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
118 entry.rulelines.append(RuleLine(line[1], False))
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
119 state = 2
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
120 elif line[0] == "allow":
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
121 if state != 0:
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
122 entry.rulelines.append(RuleLine(line[1], True))
48085
9f9e5fe63ec2 Close issue 3437 - missing state change when Allow lines are processed.
Skip Montanaro <skip@pobox.com>
parents: 47939
diff changeset
123 state = 2
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
124 if state == 2:
63506
2c253f6e02e7 Merged revisions 83449 via svnmerge from
Georg Brandl <georg@python.org>
parents: 48085
diff changeset
125 self._add_entry(entry)
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
126
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
127
12062
c43b38079228 Skip Montanaro:
Guido van Rossum <guido@python.org>
parents: 7887
diff changeset
128 def can_fetch(self, useragent, url):
c43b38079228 Skip Montanaro:
Guido van Rossum <guido@python.org>
parents: 7887
diff changeset
129 """using the parsed robots.txt decide if useragent can fetch url"""
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
130 if self.disallow_all:
22830
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
131 return False
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
132 if self.allow_all:
22830
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
133 return True
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
134 # search for given user agent matches
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
135 # the first match counts
63272
0bb47c51e933 Merged revisions 83209 via svnmerge from
Senthil Kumaran <orsenthil@gmail.com>
parents: 48085
diff changeset
136 parsed_url = urlparse.urlparse(urllib.unquote(url))
0bb47c51e933 Merged revisions 83209 via svnmerge from
Senthil Kumaran <orsenthil@gmail.com>
parents: 48085
diff changeset
137 url = urlparse.urlunparse(('', '', parsed_url.path,
0bb47c51e933 Merged revisions 83209 via svnmerge from
Senthil Kumaran <orsenthil@gmail.com>
parents: 48085
diff changeset
138 parsed_url.params, parsed_url.query, parsed_url.fragment))
0bb47c51e933 Merged revisions 83209 via svnmerge from
Senthil Kumaran <orsenthil@gmail.com>
parents: 48085
diff changeset
139 url = urllib.quote(url)
0bb47c51e933 Merged revisions 83209 via svnmerge from
Senthil Kumaran <orsenthil@gmail.com>
parents: 48085
diff changeset
140 if not url:
0bb47c51e933 Merged revisions 83209 via svnmerge from
Senthil Kumaran <orsenthil@gmail.com>
parents: 48085
diff changeset
141 url = "/"
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
142 for entry in self.entries:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
143 if entry.applies_to(useragent):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
144 return entry.allowance(url)
22088
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
145 # try the default entry last
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
146 if self.default_entry:
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
147 return self.default_entry.allowance(url)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
148 # agent not found ==> access granted
22830
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
149 return True
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
150
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
151
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
152 def __str__(self):
41146
8fa4e980b589 Patch #1555098: use str.join() instead of repeated string
Georg Brandl <georg@python.org>
parents: 33240
diff changeset
153 return ''.join([str(entry) + "\n" for entry in self.entries])
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
154
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
155
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
156 class RuleLine:
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
157 """A rule line is a single "Allow:" (allowance==True) or "Disallow:"
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
158 (allowance==False) followed by a path."""
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
159 def __init__(self, path, allowance):
22088
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
160 if path == '' and not allowance:
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
161 # an empty value means allow all
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
162 allowance = True
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
163 self.path = urllib.quote(path)
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
164 self.allowance = allowance
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
165
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
166 def applies_to(self, filename):
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
167 return self.path == "*" or filename.startswith(self.path)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
168
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
169 def __str__(self):
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
170 return (self.allowance and "Allow" or "Disallow") + ": " + self.path
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
171
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
172
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
173 class Entry:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
174 """An entry has one or more user-agents and zero or more rulelines"""
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
175 def __init__(self):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
176 self.useragents = []
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
177 self.rulelines = []
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
178
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
179 def __str__(self):
41146
8fa4e980b589 Patch #1555098: use str.join() instead of repeated string
Georg Brandl <georg@python.org>
parents: 33240
diff changeset
180 ret = []
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
181 for agent in self.useragents:
41146
8fa4e980b589 Patch #1555098: use str.join() instead of repeated string
Georg Brandl <georg@python.org>
parents: 33240
diff changeset
182 ret.extend(["User-agent: ", agent, "\n"])
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
183 for line in self.rulelines:
41146
8fa4e980b589 Patch #1555098: use str.join() instead of repeated string
Georg Brandl <georg@python.org>
parents: 33240
diff changeset
184 ret.extend([str(line), "\n"])
8fa4e980b589 Patch #1555098: use str.join() instead of repeated string
Georg Brandl <georg@python.org>
parents: 33240
diff changeset
185 return ''.join(ret)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
186
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
187 def applies_to(self, useragent):
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
188 """check if this entry applies to the specified agent"""
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
189 # split the name token and make it lower case
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
190 useragent = useragent.split("/")[0].lower()
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
191 for agent in self.useragents:
46591
1ea4d1b18aed Get rid of _test(), _main(), _debug() and _check(). Tests are no longer
Skip Montanaro <skip@pobox.com>
parents: 42956
diff changeset
192 if agent == '*':
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
193 # we have the catch-all agent
22830
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
194 return True
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
195 agent = agent.lower()
31967
3d046d063a84 Replace str.find()!=1 with the more readable "in" operator.
Raymond Hettinger <python@rcn.com>
parents: 31653
diff changeset
196 if agent in useragent:
22830
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
197 return True
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
198 return False
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
199
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
200 def allowance(self, filename):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
201 """Preconditions:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
202 - our agent applies to this entry
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
203 - filename is URL decoded"""
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
204 for line in self.rulelines:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
205 if line.applies_to(filename):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
206 return line.allowance
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
207 return True
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
208
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
209 class URLopener(urllib.FancyURLopener):
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
210 def __init__(self, *args):
28089
14f65d661084 Get rid of many apply() calls.
Guido van Rossum <guido@python.org>
parents: 23591
diff changeset
211 urllib.FancyURLopener.__init__(self, *args)
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
212 self.errcode = 200
16616
e2a63584566f Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16579
diff changeset
213
42957
533fb5321d6b backport r57626
Skip Montanaro <skip@pobox.com>
parents: 33240
diff changeset
214 def prompt_user_passwd(self, host, realm):
533fb5321d6b backport r57626
Skip Montanaro <skip@pobox.com>
parents: 33240
diff changeset
215 ## If robots.txt file is accessible only with a password,
533fb5321d6b backport r57626
Skip Montanaro <skip@pobox.com>
parents: 33240
diff changeset
216 ## we act as if the file wasn't there.
533fb5321d6b backport r57626
Skip Montanaro <skip@pobox.com>
parents: 33240
diff changeset
217 return None, None
533fb5321d6b backport r57626
Skip Montanaro <skip@pobox.com>
parents: 33240
diff changeset
218
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
219 def http_error_default(self, url, fp, errcode, errmsg, headers):
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
220 self.errcode = errcode
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
221 return urllib.FancyURLopener.http_error_default(self, url, fp, errcode,
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
222 errmsg, headers)