Mercurial > cpython-lru_cache-weakref

annotate Lib/robotparser.py @ 16413:8819867f14ef legacy-trunk

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Patch derived from Trent's 101162: a Python/C API testing framework. STILL NEEDS UNIX BUILD CHANGES.
author Tim Peters <tim.peters@gmail.com>
date Fri, 02 Feb 2001 05:57:15 +0000
parents 7e55fa93f4db
children f4d15fdd15dd
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
1 """ robotparser.py
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
2
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
3 Copyright (C) 2000 Bastian Kleineidam
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
4
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
5 You can choose between two licenses when using this package:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
6 1) GNU GPLv2
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
7 2) PYTHON 2.0 OPEN SOURCE LICENSE
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
8
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
9 The robots.txt Exclusion Protocol is implemented as specified in
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
10 http://info.webcrawler.com/mak/projects/robots/norobots-rfc.html
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
11 """
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
12 import re,string,urlparse,urllib
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
13
16070
b5345e99596b added __all__ lists to a number of Python modules
Skip Montanaro <skip@pobox.com>
parents: 16067
diff changeset
14 __all__ = ["RobotFileParser"]
b5345e99596b added __all__ lists to a number of Python modules
Skip Montanaro <skip@pobox.com>
parents: 16067
diff changeset
15
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
16 debug = 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
17
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
18 def _debug(msg):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
19 if debug: print msg
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
20
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
21
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
22 class RobotFileParser:
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
23 def __init__(self, url=''):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
24 self.entries = []
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
25 self.disallow_all = 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
26 self.allow_all = 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
27 self.set_url(url)
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
28 self.last_checked = 0
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
29
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
30 def mtime(self):
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
31 return self.last_checked
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
32
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
33 def modified(self):
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
34 import time
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
35 self.last_checked = time.time()
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
36
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
37 def set_url(self, url):
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
38 self.url = url
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
39 self.host, self.path = urlparse.urlparse(url)[1:3]
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
40
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
41 def read(self):
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
42 import httplib
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
43 tries = 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
44 while tries<5:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
45 connection = httplib.HTTP(self.host)
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
46 connection.putrequest("GET", self.path)
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
47 connection.putheader("Host", self.host)
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
48 connection.endheaders()
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
49 status, text, mime = connection.getreply()
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
50 if status in [301,302] and mime:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
51 tries = tries + 1
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
52 newurl = mime.get("Location", mime.get("Uri", ""))
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
53 newurl = urlparse.urljoin(self.url, newurl)
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
54 self.set_url(newurl)
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
55 else:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
56 break
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
57 if status==401 or status==403:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
58 self.disallow_all = 1
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
59 elif status>=400:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
60 self.allow_all = 1
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
61 else:
16074
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
62 # status < 400
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
63 self.parse(connection.getfile().readlines())
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
64
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
65 def parse(self, lines):
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
66 """parse the input lines from a robot.txt file.
16074
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
67 We allow that a user-agent: line is not preceded by
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
68 one or more blank lines."""
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
69 state = 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
70 linenumber = 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
71 entry = Entry()
16074
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
72
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
73 for line in lines:
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
74 line = string.strip(line)
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
75 linenumber = linenumber + 1
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
76 if not line:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
77 if state==1:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
78 _debug("line %d: warning: you should insert"
16074
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
79 " allow: or disallow: directives below any"
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
80 " user-agent: line" % linenumber)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
81 entry = Entry()
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
82 state = 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
83 elif state==2:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
84 self.entries.append(entry)
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
85 entry = Entry()
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
86 state = 0
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
87 # remove optional comment and strip line
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
88 i = string.find(line, '#')
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
89 if i>=0:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
90 line = line[:i]
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
91 line = string.strip(line)
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
92 if not line:
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
93 continue
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
94 line = string.split(line, ':', 1)
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
95 if len(line) == 2:
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
96 line[0] = string.lower(string.strip(line[0]))
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
97 line[1] = string.strip(line[1])
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
98 if line[0] == "user-agent":
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
99 if state==2:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
100 _debug("line %d: warning: you should insert a blank"
16074
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
101 " line before any user-agent"
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
102 " directive" % linenumber)
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
103 self.entries.append(entry)
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
104 entry = Entry()
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
105 entry.useragents.append(line[1])
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
106 state = 1
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
107 elif line[0] == "disallow":
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
108 if state==0:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
109 _debug("line %d: error: you must insert a user-agent:"
16074
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
110 " directive before this line" % linenumber)
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
111 else:
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
112 entry.rulelines.append(RuleLine(line[1], 0))
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
113 state = 2
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
114 elif line[0] == "allow":
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
115 if state==0:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
116 _debug("line %d: error: you must insert a user-agent:"
16074
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
117 " directive before this line" % linenumber)
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
118 else:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
119 entry.rulelines.append(RuleLine(line[1], 1))
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
120 else:
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
121 _debug("line %d: warning: unknown key %s" % (linenumber,
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
122 line[0]))
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
123 else:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
124 _debug("line %d: error: malformed line %s"%(linenumber, line))
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
125 if state==2:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
126 self.entries.append(entry)
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
127 _debug("Parsed rules:\n%s" % str(self))
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
128
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
129
12062
c43b38079228 Skip Montanaro:
Guido van Rossum <guido@python.org>
parents: 7887
diff changeset
130 def can_fetch(self, useragent, url):
c43b38079228 Skip Montanaro:
Guido van Rossum <guido@python.org>
parents: 7887
diff changeset
131 """using the parsed robots.txt decide if useragent can fetch url"""
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
132 _debug("Checking robot.txt allowance for\n%s\n%s" % (useragent, url))
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
133 if self.disallow_all:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
134 return 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
135 if self.allow_all:
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
136 return 1
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
137 # search for given user agent matches
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
138 # the first match counts
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
139 useragent = string.lower(useragent)
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
140 url = urllib.quote(urlparse.urlparse(url)[2])
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
141 for entry in self.entries:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
142 if entry.applies_to(useragent):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
143 return entry.allowance(url)
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
144 # agent not found ==> access granted
7887
efbb7e57e6f8 Give in to tabnanny
Guido van Rossum <guido@python.org>
parents: 5063
diff changeset
145 return 1
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
146
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
147
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
148 def __str__(self):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
149 ret = ""
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
150 for entry in self.entries:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
151 ret = ret + str(entry) + "\n"
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
152 return ret
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
153
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
154
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
155 class RuleLine:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
156 """A rule line is a single "Allow:" (allowance==1) or "Disallow:"
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
157 (allowance==0) followed by a path."""
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
158 def __init__(self, path, allowance):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
159 self.path = urllib.quote(path)
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
160 self.allowance = allowance
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
161
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
162 def applies_to(self, filename):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
163 return self.path=="*" or re.match(self.path, filename)
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
164
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
165 def __str__(self):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
166 return (self.allowance and "Allow" or "Disallow")+": "+self.path
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
167
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
168
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
169 class Entry:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
170 """An entry has one or more user-agents and zero or more rulelines"""
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
171 def __init__(self):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
172 self.useragents = []
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
173 self.rulelines = []
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
174
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
175 def __str__(self):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
176 ret = ""
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
177 for agent in self.useragents:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
178 ret = ret + "User-agent: "+agent+"\n"
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
179 for line in self.rulelines:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
180 ret = ret + str(line) + "\n"
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
181 return ret
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
182
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
183 def applies_to(self, useragent):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
184 "check if this entry applies to the specified agent"
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
185 for agent in self.useragents:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
186 if agent=="*":
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
187 return 1
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
188 if re.match(agent, useragent):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
189 return 1
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
190 return 0
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
191
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
192 def allowance(self, filename):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
193 """Preconditions:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
194 - our agent applies to this entry
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
195 - filename is URL decoded"""
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
196 for line in self.rulelines:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
197 if line.applies_to(filename):
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
198 return line.allowance
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
199 return 1
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
200
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
201
12062
c43b38079228 Skip Montanaro:
Guido van Rossum <guido@python.org>
parents: 7887
diff changeset
202 def _test():
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
203 global debug
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
204 import sys
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
205 rp = RobotFileParser()
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
206 debug = 1
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
207 if len(sys.argv) <= 1:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
208 rp.set_url('http://www.musi-cal.com/robots.txt')
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
209 rp.read()
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
210 else:
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
211 rp.parse(open(sys.argv[1]).readlines())
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
212 print rp.can_fetch('*', 'http://www.musi-cal.com/')
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
213 print rp.can_fetch('Musi-Cal-Robot/1.0',
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
214 'http://www.musi-cal.com/cgi-bin/event-search'
16074
7e55fa93f4db Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
215 '?city=San+Francisco')
5063
f0731fa4447f Skip Montanaro's robots.txt parser.
Guido van Rossum <guido@python.org>
parents:
diff changeset
216
16067
fe36ab4b3e97 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229.
Skip Montanaro <skip@pobox.com>
parents: 12062
diff changeset
217 if __name__ == '__main__':
12062
c43b38079228 Skip Montanaro:
Guido van Rossum <guido@python.org>
parents: 7887
diff changeset
218 _test()