Mercurial > cpython-lru_cache-weakref

annotate Lib/robotparser.py @ 47426:4dedef4c0dfd

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Blocked revisions 63863 via svnmerge ........ r63863 | benjamin.peterson | 2008-06-01 14:01:25 -0500 (Sun, 01 Jun 2008) | 2 lines add a warning about bgen being removed ........
author Benjamin Peterson <benjamin@python.org>
date Sun, 01 Jun 2008 19:04:01 +0000
parents 031dfe455585
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
1 """ robotparser.py
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
2
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
3 Copyright (C) 2000 Bastian Kleineidam
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
4
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
5 You can choose between two licenses when using this package:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
6 1) GNU GPLv2
22370
c210562e1558 Patch #499513: use readline() instead of readlines(). Removed the
Martin v. Löwis <martin@v.loewis.de>
parents: 22088
diff changeset
7 2) PSF license for Python 2.2
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
8
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
9 The robots.txt Exclusion Protocol is implemented as specified in
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
10 http://info.webcrawler.com/mak/projects/robots/norobots-rfc.html
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
11 """
46664
031dfe455585 Merged revisions 62425-62429,62434-62436,62441,62444,62446-62448,62450-62455,62463,62465-62466,62469,62474,62476-62478,62480,62485,62492,62497-62498,62500,62507,62513-62514,62516,62521,62531,62535,62545-62546,62548-62551,62553-62559,62569,62574,62577,62593,62595,62604-62606,62608,62616,62626-62627,62636,62638,62644-62645,62647-62648,62651-62653,62656,62661,62663,62680,62686-62687,62696,62699-62703,62711 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 43095
diff changeset
12 import urlparse
031dfe455585 Merged revisions 62425-62429,62434-62436,62441,62444,62446-62448,62450-62455,62463,62465-62466,62469,62474,62476-62478,62480,62485,62492,62497-62498,62500,62507,62513-62514,62516,62521,62531,62535,62545-62546,62548-62551,62553-62559,62569,62574,62577,62593,62595,62604-62606,62608,62616,62626-62627,62636,62638,62644-62645,62647-62648,62651-62653,62656,62661,62663,62680,62686-62687,62696,62699-62703,62711 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 43095
diff changeset
13 import urllib
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
14
16070
b5345e99596b added __all__ lists to a number of Python modules
Skip Montanaro <skip@pobox.com>
parents: 16067
diff changeset
15 __all__ = ["RobotFileParser"]
b5345e99596b added __all__ lists to a number of Python modules
Skip Montanaro <skip@pobox.com>
parents: 16067
diff changeset
16
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
17 class RobotFileParser:
23565
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
18 """ This class provides a set of methods to read, parse and answer
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
19 questions about a single robots.txt file.
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
20
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
21 """
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
22
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
23 def __init__(self, url=''):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
24 self.entries = []
22088
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
25 self.default_entry = None
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
26 self.disallow_all = False
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
27 self.allow_all = False
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
28 self.set_url(url)
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
29 self.last_checked = 0
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
30
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
31 def mtime(self):
23565
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
32 """Returns the time the robots.txt file was last fetched.
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
33
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
34 This is useful for long-running web spiders that need to
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
35 check for new robots.txt files periodically.
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
36
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
37 """
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
38 return self.last_checked
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
39
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
40 def modified(self):
23565
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
41 """Sets the time the robots.txt file was last fetched to the
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
42 current time.
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
43
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
44 """
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
45 import time
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
46 self.last_checked = time.time()
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
47
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
48 def set_url(self, url):
23565
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
49 """Sets the URL referring to a robots.txt file."""
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
50 self.url = url
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
51 self.host, self.path = urlparse.urlparse(url)[1:3]
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
52
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
53 def read(self):
23565
1a67fbc7a066 Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2).
Raymond Hettinger <python@rcn.com>
parents: 22830
diff changeset
54 """Reads the robots.txt URL and feeds it to the parser."""
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
55 opener = URLopener()
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
56 f = opener.open(self.url)
22370
c210562e1558 Patch #499513: use readline() instead of readlines(). Removed the
Martin v. Löwis <martin@v.loewis.de>
parents: 22088
diff changeset
57 lines = []
c210562e1558 Patch #499513: use readline() instead of readlines(). Removed the
Martin v. Löwis <martin@v.loewis.de>
parents: 22088
diff changeset
58 line = f.readline()
c210562e1558 Patch #499513: use readline() instead of readlines(). Removed the
Martin v. Löwis <martin@v.loewis.de>
parents: 22088
diff changeset
59 while line:
c210562e1558 Patch #499513: use readline() instead of readlines(). Removed the
Martin v. Löwis <martin@v.loewis.de>
parents: 22088
diff changeset
60 lines.append(line.strip())
c210562e1558 Patch #499513: use readline() instead of readlines(). Removed the
Martin v. Löwis <martin@v.loewis.de>
parents: 22088
diff changeset
61 line = f.readline()
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
62 self.errcode = opener.errcode
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
63 if self.errcode in (401, 403):
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
64 self.disallow_all = True
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
65 elif self.errcode >= 400:
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
66 self.allow_all = True
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
67 elif self.errcode == 200 and lines:
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
68 self.parse(lines)
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
69
22371
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
70 def _add_entry(self, entry):
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
71 if "*" in entry.useragents:
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
72 # the default entry is considered last
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
73 self.default_entry = entry
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
74 else:
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
75 self.entries.append(entry)
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
76
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
77 def parse(self, lines):
31653
f2600448e85c SF patch #911431: robot.txt must be robots.txt
Raymond Hettinger <python@rcn.com>
parents: 28089
diff changeset
78 """parse the input lines from a robots.txt file.
16074
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
79 We allow that a user-agent: line is not preceded by
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
80 one or more blank lines."""
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
81 state = 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
82 linenumber = 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
83 entry = Entry()
16074
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
84
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
85 for line in lines:
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
86 linenumber = linenumber + 1
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
87 if not line:
46664
031dfe455585 Merged revisions 62425-62429,62434-62436,62441,62444,62446-62448,62450-62455,62463,62465-62466,62469,62474,62476-62478,62480,62485,62492,62497-62498,62500,62507,62513-62514,62516,62521,62531,62535,62545-62546,62548-62551,62553-62559,62569,62574,62577,62593,62595,62604-62606,62608,62616,62626-62627,62636,62638,62644-62645,62647-62648,62651-62653,62656,62661,62663,62680,62686-62687,62696,62699-62703,62711 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 43095
diff changeset
88 if state == 1:
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
89 entry = Entry()
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
90 state = 0
46664
031dfe455585 Merged revisions 62425-62429,62434-62436,62441,62444,62446-62448,62450-62455,62463,62465-62466,62469,62474,62476-62478,62480,62485,62492,62497-62498,62500,62507,62513-62514,62516,62521,62531,62535,62545-62546,62548-62551,62553-62559,62569,62574,62577,62593,62595,62604-62606,62608,62616,62626-62627,62636,62638,62644-62645,62647-62648,62651-62653,62656,62661,62663,62680,62686-62687,62696,62699-62703,62711 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 43095
diff changeset
91 elif state == 2:
22371
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
92 self._add_entry(entry)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
93 entry = Entry()
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
94 state = 0
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
95 # remove optional comment and strip line
16494
f4d15fdd15dd String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents: 16074
diff changeset
96 i = line.find('#')
46664
031dfe455585 Merged revisions 62425-62429,62434-62436,62441,62444,62446-62448,62450-62455,62463,62465-62466,62469,62474,62476-62478,62480,62485,62492,62497-62498,62500,62507,62513-62514,62516,62521,62531,62535,62545-62546,62548-62551,62553-62559,62569,62574,62577,62593,62595,62604-62606,62608,62616,62626-62627,62636,62638,62644-62645,62647-62648,62651-62653,62656,62661,62663,62680,62686-62687,62696,62699-62703,62711 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 43095
diff changeset
97 if i >= 0:
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
98 line = line[:i]
16494
f4d15fdd15dd String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents: 16074
diff changeset
99 line = line.strip()
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
100 if not line:
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
101 continue
16494
f4d15fdd15dd String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents: 16074
diff changeset
102 line = line.split(':', 1)
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
103 if len(line) == 2:
16494
f4d15fdd15dd String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents: 16074
diff changeset
104 line[0] = line[0].strip().lower()
22088
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
105 line[1] = urllib.unquote(line[1].strip())
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
106 if line[0] == "user-agent":
46664
031dfe455585 Merged revisions 62425-62429,62434-62436,62441,62444,62446-62448,62450-62455,62463,62465-62466,62469,62474,62476-62478,62480,62485,62492,62497-62498,62500,62507,62513-62514,62516,62521,62531,62535,62545-62546,62548-62551,62553-62559,62569,62574,62577,62593,62595,62604-62606,62608,62616,62626-62627,62636,62638,62644-62645,62647-62648,62651-62653,62656,62661,62663,62680,62686-62687,62696,62699-62703,62711 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 43095
diff changeset
107 if state == 2:
22371
b357df301f5f Correctly set default entry in all cases.
Martin v. Löwis <martin@v.loewis.de>
parents: 22370
diff changeset
108 self._add_entry(entry)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
109 entry = Entry()
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
110 entry.useragents.append(line[1])
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
111 state = 1
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
112 elif line[0] == "disallow":
46664
031dfe455585 Merged revisions 62425-62429,62434-62436,62441,62444,62446-62448,62450-62455,62463,62465-62466,62469,62474,62476-62478,62480,62485,62492,62497-62498,62500,62507,62513-62514,62516,62521,62531,62535,62545-62546,62548-62551,62553-62559,62569,62574,62577,62593,62595,62604-62606,62608,62616,62626-62627,62636,62638,62644-62645,62647-62648,62651-62653,62656,62661,62663,62680,62686-62687,62696,62699-62703,62711 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 43095
diff changeset
113 if state != 0:
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
114 entry.rulelines.append(RuleLine(line[1], False))
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
115 state = 2
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
116 elif line[0] == "allow":
46664
031dfe455585 Merged revisions 62425-62429,62434-62436,62441,62444,62446-62448,62450-62455,62463,62465-62466,62469,62474,62476-62478,62480,62485,62492,62497-62498,62500,62507,62513-62514,62516,62521,62531,62535,62545-62546,62548-62551,62553-62559,62569,62574,62577,62593,62595,62604-62606,62608,62616,62626-62627,62636,62638,62644-62645,62647-62648,62651-62653,62656,62661,62663,62680,62686-62687,62696,62699-62703,62711 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 43095
diff changeset
117 if state != 0:
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
118 entry.rulelines.append(RuleLine(line[1], True))
46664
031dfe455585 Merged revisions 62425-62429,62434-62436,62441,62444,62446-62448,62450-62455,62463,62465-62466,62469,62474,62476-62478,62480,62485,62492,62497-62498,62500,62507,62513-62514,62516,62521,62531,62535,62545-62546,62548-62551,62553-62559,62569,62574,62577,62593,62595,62604-62606,62608,62616,62626-62627,62636,62638,62644-62645,62647-62648,62651-62653,62656,62661,62663,62680,62686-62687,62696,62699-62703,62711 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 43095
diff changeset
119 if state == 2:
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
120 self.entries.append(entry)
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
121
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
122
12062
c43b38079228 Skip Montanaro:
Guido van Rossum <guido@python.org>
parents: 7887
diff changeset
123 def can_fetch(self, useragent, url):
c43b38079228 Skip Montanaro:
Guido van Rossum <guido@python.org>
parents: 7887
diff changeset
124 """using the parsed robots.txt decide if useragent can fetch url"""
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
125 if self.disallow_all:
22830
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
126 return False
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
127 if self.allow_all:
22830
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
128 return True
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
129 # search for given user agent matches
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
130 # the first match counts
22088
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
131 url = urllib.quote(urlparse.urlparse(urllib.unquote(url))[2]) or "/"
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
132 for entry in self.entries:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
133 if entry.applies_to(useragent):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
134 return entry.allowance(url)
22088
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
135 # try the default entry last
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
136 if self.default_entry:
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
137 return self.default_entry.allowance(url)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
138 # agent not found ==> access granted
22830
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
139 return True
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
140
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
141
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
142 def __str__(self):
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
143 return ''.join([str(entry) + "\n" for entry in self.entries])
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
144
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
145
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
146 class RuleLine:
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
147 """A rule line is a single "Allow:" (allowance==True) or "Disallow:"
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
148 (allowance==False) followed by a path."""
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
149 def __init__(self, path, allowance):
22088
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
150 if path == '' and not allowance:
736fb5aec9b3 Correct various errors:
Martin v. Löwis <martin@v.loewis.de>
parents: 19213
diff changeset
151 # an empty value means allow all
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
152 allowance = True
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
153 self.path = urllib.quote(path)
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
154 self.allowance = allowance
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
155
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
156 def applies_to(self, filename):
46664
031dfe455585 Merged revisions 62425-62429,62434-62436,62441,62444,62446-62448,62450-62455,62463,62465-62466,62469,62474,62476-62478,62480,62485,62492,62497-62498,62500,62507,62513-62514,62516,62521,62531,62535,62545-62546,62548-62551,62553-62559,62569,62574,62577,62593,62595,62604-62606,62608,62616,62626-62627,62636,62638,62644-62645,62647-62648,62651-62653,62656,62661,62663,62680,62686-62687,62696,62699-62703,62711 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 43095
diff changeset
157 return self.path == "*" or filename.startswith(self.path)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
158
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
159 def __str__(self):
46664
031dfe455585 Merged revisions 62425-62429,62434-62436,62441,62444,62446-62448,62450-62455,62463,62465-62466,62469,62474,62476-62478,62480,62485,62492,62497-62498,62500,62507,62513-62514,62516,62521,62531,62535,62545-62546,62548-62551,62553-62559,62569,62574,62577,62593,62595,62604-62606,62608,62616,62626-62627,62636,62638,62644-62645,62647-62648,62651-62653,62656,62661,62663,62680,62686-62687,62696,62699-62703,62711 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 43095
diff changeset
160 return (self.allowance and "Allow" or "Disallow") + ": " + self.path
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
161
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
162
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
163 class Entry:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
164 """An entry has one or more user-agents and zero or more rulelines"""
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
165 def __init__(self):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
166 self.useragents = []
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
167 self.rulelines = []
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
168
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
169 def __str__(self):
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
170 ret = []
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
171 for agent in self.useragents:
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
172 ret.extend(["User-agent: ", agent, "\n"])
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
173 for line in self.rulelines:
41554
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
174 ret.extend([str(line), "\n"])
06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from
Guido van Rossum <guido@python.org>
parents: 40775
diff changeset
175 return ''.join(ret)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
176
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
177 def applies_to(self, useragent):
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
178 """check if this entry applies to the specified agent"""
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
179 # split the name token and make it lower case
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
180 useragent = useragent.split("/")[0].lower()
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
181 for agent in self.useragents:
46664
031dfe455585 Merged revisions 62425-62429,62434-62436,62441,62444,62446-62448,62450-62455,62463,62465-62466,62469,62474,62476-62478,62480,62485,62492,62497-62498,62500,62507,62513-62514,62516,62521,62531,62535,62545-62546,62548-62551,62553-62559,62569,62574,62577,62593,62595,62604-62606,62608,62616,62626-62627,62636,62638,62644-62645,62647-62648,62651-62653,62656,62661,62663,62680,62686-62687,62696,62699-62703,62711 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 43095
diff changeset
182 if agent == '*':
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
183 # we have the catch-all agent
22830
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
184 return True
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
185 agent = agent.lower()
31967
3d046d063a84 Replace str.find()!=1 with the more readable "in" operator.
Raymond Hettinger <python@rcn.com>
parents: 31653
diff changeset
186 if agent in useragent:
22830
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
187 return True
a9dd5e6d6805 Convert a pile of obvious "yes/no" functions to return bool.
Tim Peters <tim.peters@gmail.com>
parents: 22371
diff changeset
188 return False
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
189
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
190 def allowance(self, filename):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
191 """Preconditions:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
192 - our agent applies to this entry
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
193 - filename is URL decoded"""
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
194 for line in self.rulelines:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
195 if line.applies_to(filename):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
196 return line.allowance
33240
71427915796e Patch #1014237: Consistently return booleans throughout.
Martin v. Löwis <martin@v.loewis.de>
parents: 31967
diff changeset
197 return True
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
198
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
199 class URLopener(urllib.FancyURLopener):
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
200 def __init__(self, *args):
28089
14f65d661084 Get rid of many apply() calls.
Guido van Rossum <guido@python.org>
parents: 23591
diff changeset
201 urllib.FancyURLopener.__init__(self, *args)
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
202 self.errcode = 200
16616
e2a63584566f Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16579
diff changeset
203
43095
4a07a288f5c4 Merged revisions 57620-57771 via svnmerge from
Thomas Wouters <thomas@python.org>
parents: 41554
diff changeset
204 def prompt_user_passwd(self, host, realm):
4a07a288f5c4 Merged revisions 57620-57771 via svnmerge from
Thomas Wouters <thomas@python.org>
parents: 41554
diff changeset
205 ## If robots.txt file is accessible only with a password,
4a07a288f5c4 Merged revisions 57620-57771 via svnmerge from
Thomas Wouters <thomas@python.org>
parents: 41554
diff changeset
206 ## we act as if the file wasn't there.
4a07a288f5c4 Merged revisions 57620-57771 via svnmerge from
Thomas Wouters <thomas@python.org>
parents: 41554
diff changeset
207 return None, None
4a07a288f5c4 Merged revisions 57620-57771 via svnmerge from
Thomas Wouters <thomas@python.org>
parents: 41554
diff changeset
208
16579
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
209 def http_error_default(self, url, fp, errcode, errmsg, headers):
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
210 self.errcode = errcode
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
211 return urllib.FancyURLopener.http_error_default(self, url, fp, errcode,
a63dc59be0f9 The bulk of the credit for these changes goes to Bastian Kleineidam
Skip Montanaro <skip@pobox.com>
parents: 16494
diff changeset
212 errmsg, headers)