cpython-lru_cache-weakref: Lib/test/test

annotate Lib/test/test_robotparser.py @ 70817:dd775b1561ba

Fix markup.

author	Georg Brandl <georg@python.org>
date	Tue, 14 Jun 2011 21:09:55 +0200
parents	3c9232a684ae
children	6e72490bbff6

rev	line source
42501 f6069f9dd52d SF patch# 1770008 by Christian Heimes (plus some extras). Guido van Rossum <guido@python.org> parents: 41554 diff changeset	1 import io
48334 5a416a6417d3 Make a new urllib package . Jeremy Hylton <jeremy@alum.mit.edu> parents: 47201 diff changeset	2 import unittest
5a416a6417d3 Make a new urllib package . Jeremy Hylton <jeremy@alum.mit.edu> parents: 47201 diff changeset	3 import urllib.robotparser
60202 5496503b8b1a Merged revision 79605 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 55593 diff changeset	4 from urllib.error import URLError
47201 9c930a32cebb #2621 rename test.test_support to test.support Benjamin Peterson <benjamin@python.org> parents: 43095 diff changeset	5 from test import support
22088 736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	6
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	7 class RobotTestCase(unittest.TestCase):
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	8 def __init__(self, index, parser, url, good, agent):
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	9 unittest.TestCase.__init__(self)
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	10 if good:
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	11 self.str = "RobotTest(%d, good, %s)" % (index, url)
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	12 else:
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	13 self.str = "RobotTest(%d, bad, %s)" % (index, url)
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	14 self.parser = parser
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	15 self.url = url
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	16 self.good = good
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	17 self.agent = agent
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	18
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	19 def runTest(self):
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	20 if isinstance(self.url, tuple):
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	21 agent, url = self.url
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	22 else:
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	23 url = self.url
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	24 agent = self.agent
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	25 if self.good:
55593 0bb232078b8c Merged revisions 73715 via svnmerge from Georg Brandl <georg@python.org> parents: 48594 diff changeset	26 self.assertTrue(self.parser.can_fetch(agent, url))
22088 736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	27 else:
55593 0bb232078b8c Merged revisions 73715 via svnmerge from Georg Brandl <georg@python.org> parents: 48594 diff changeset	28 self.assertFalse(self.parser.can_fetch(agent, url))
22088 736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	29
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	30 def __str__(self):
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	31 return self.str
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	32
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	33 tests = unittest.TestSuite()
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	34
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	35 def RobotTest(index, robots_txt, good_urls, bad_urls,
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	36 agent="test_robotparser"):
23075 b7e87732f6ab Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 22088 diff changeset	37
42501 f6069f9dd52d SF patch# 1770008 by Christian Heimes (plus some extras). Guido van Rossum <guido@python.org> parents: 41554 diff changeset	38 lines = io.StringIO(robots_txt).readlines()
48334 5a416a6417d3 Make a new urllib package . Jeremy Hylton <jeremy@alum.mit.edu> parents: 47201 diff changeset	39 parser = urllib.robotparser.RobotFileParser()
23075 b7e87732f6ab Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 22088 diff changeset	40 parser.parse(lines)
b7e87732f6ab Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 22088 diff changeset	41 for url in good_urls:
b7e87732f6ab Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 22088 diff changeset	42 tests.addTest(RobotTestCase(index, parser, url, 1, agent))
b7e87732f6ab Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 22088 diff changeset	43 for url in bad_urls:
b7e87732f6ab Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 22088 diff changeset	44 tests.addTest(RobotTestCase(index, parser, url, 0, agent))
22088 736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	45
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	46 # Examples from http://www.robotstxt.org/wc/norobots.html (fetched 2002)
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	47
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	48 # 1.
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	49 doc = """
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	50 User-agent: *
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	51 Disallow: /cyberworld/map/ # This is an infinite virtual URL space
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	52 Disallow: /tmp/ # these will soon disappear
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	53 Disallow: /foo.html
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	54 """
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	55
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	56 good = ['/','/test.html']
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	57 bad = ['/cyberworld/map/index.html','/tmp/xxx','/foo.html']
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	58
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	59 RobotTest(1, doc, good, bad)
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	60
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	61 # 2.
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	62 doc = """
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	63 # robots.txt for http://www.example.com/
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	64
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	65 User-agent: *
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	66 Disallow: /cyberworld/map/ # This is an infinite virtual URL space
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	67
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	68 # Cybermapper knows where to go.
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	69 User-agent: cybermapper
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	70 Disallow:
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	71
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	72 """
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	73
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	74 good = ['/','/test.html',('cybermapper','/cyberworld/map/index.html')]
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	75 bad = ['/cyberworld/map/index.html']
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	76
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	77 RobotTest(2, doc, good, bad)
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	78
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	79 # 3.
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	80 doc = """
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	81 # go away
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	82 User-agent: *
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	83 Disallow: /
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	84 """
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	85
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	86 good = []
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	87 bad = ['/cyberworld/map/index.html','/','/tmp/']
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	88
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	89 RobotTest(3, doc, good, bad)
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	90
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	91 # Examples from http://www.robotstxt.org/wc/norobots-rfc.html (fetched 2002)
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	92
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	93 # 4.
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	94 doc = """
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	95 User-agent: figtree
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	96 Disallow: /tmp
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	97 Disallow: /a%3cd.html
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	98 Disallow: /a%2fb.html
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	99 Disallow: /%7ejoe/index.html
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	100 """
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	101
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	102 good = [] # XFAIL '/a/b.html'
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	103 bad = ['/tmp','/tmp.html','/tmp/a.html',
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	104 '/a%3cd.html','/a%3Cd.html','/a%2fb.html',
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	105 '/~joe/index.html'
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	106 ]
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	107
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	108 RobotTest(4, doc, good, bad, 'figtree')
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	109 RobotTest(5, doc, good, bad, 'FigTree Robot libwww-perl/5.04')
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	110
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	111 # 6.
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	112 doc = """
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	113 User-agent: *
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	114 Disallow: /tmp/
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	115 Disallow: /a%3Cd.html
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	116 Disallow: /a/b.html
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	117 Disallow: /%7ejoe/index.html
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	118 """
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	119
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	120 good = ['/tmp',] # XFAIL: '/a%2fb.html'
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	121 bad = ['/tmp/','/tmp/a.html',
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	122 '/a%3cd.html','/a%3Cd.html',"/a/b.html",
23075 b7e87732f6ab Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 22088 diff changeset	123 '/%7Ejoe/index.html']
22088 736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	124
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	125 RobotTest(6, doc, good, bad)
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	126
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	127 # From bug report #523041
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	128
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	129 # 7.
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	130 doc = """
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	131 User-Agent: *
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	132 Disallow: /.
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	133 """
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	134
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	135 good = ['/foo.html']
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	136 bad = [] # Bug report says "/" should be denied, but that is not in the RFC
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	137
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	138 RobotTest(7, doc, good, bad)
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	139
48594 35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	140 # From Google: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40364
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	141
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	142 # 8.
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	143 doc = """
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	144 User-agent: Googlebot
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	145 Allow: /folder1/myfile.html
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	146 Disallow: /folder1/
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	147 """
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	148
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	149 good = ['/folder1/myfile.html']
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	150 bad = ['/folder1/anotherfile.html']
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	151
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	152 RobotTest(8, doc, good, bad, agent="Googlebot")
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	153
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	154 # 9. This file is incorrect because "Googlebot" is a substring of
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	155 # "Googlebot-Mobile", so test 10 works just like test 9.
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	156 doc = """
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	157 User-agent: Googlebot
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	158 Disallow: /
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	159
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	160 User-agent: Googlebot-Mobile
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	161 Allow: /
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	162 """
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	163
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	164 good = []
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	165 bad = ['/something.jpg']
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	166
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	167 RobotTest(9, doc, good, bad, agent="Googlebot")
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	168
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	169 good = []
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	170 bad = ['/something.jpg']
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	171
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	172 RobotTest(10, doc, good, bad, agent="Googlebot-Mobile")
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	173
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	174 # 11. Get the order correct.
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	175 doc = """
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	176 User-agent: Googlebot-Mobile
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	177 Allow: /
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	178
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	179 User-agent: Googlebot
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	180 Disallow: /
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	181 """
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	182
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	183 good = []
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	184 bad = ['/something.jpg']
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	185
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	186 RobotTest(11, doc, good, bad, agent="Googlebot")
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	187
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	188 good = ['/something.jpg']
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	189 bad = []
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	190
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	191 RobotTest(12, doc, good, bad, agent="Googlebot-Mobile")
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	192
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	193
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	194 # 13. Google also got the order wrong in #8. You need to specify the
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	195 # URLs from more specific to more general.
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	196 doc = """
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	197 User-agent: Googlebot
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	198 Allow: /folder1/myfile.html
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	199 Disallow: /folder1/
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	200 """
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	201
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	202 good = ['/folder1/myfile.html']
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	203 bad = ['/folder1/anotherfile.html']
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	204
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	205 RobotTest(13, doc, good, bad, agent="googlebot")
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	206
35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	207
63271 3119c4d7276b Merged revisions 83209 via svnmerge from Senthil Kumaran <orsenthil@gmail.com> parents: 60202 diff changeset	208 # 14. For issue #6325 (query string support)
3119c4d7276b Merged revisions 83209 via svnmerge from Senthil Kumaran <orsenthil@gmail.com> parents: 60202 diff changeset	209 doc = """
3119c4d7276b Merged revisions 83209 via svnmerge from Senthil Kumaran <orsenthil@gmail.com> parents: 60202 diff changeset	210 User-agent: *
3119c4d7276b Merged revisions 83209 via svnmerge from Senthil Kumaran <orsenthil@gmail.com> parents: 60202 diff changeset	211 Disallow: /some/path?name=value
3119c4d7276b Merged revisions 83209 via svnmerge from Senthil Kumaran <orsenthil@gmail.com> parents: 60202 diff changeset	212 """
3119c4d7276b Merged revisions 83209 via svnmerge from Senthil Kumaran <orsenthil@gmail.com> parents: 60202 diff changeset	213
3119c4d7276b Merged revisions 83209 via svnmerge from Senthil Kumaran <orsenthil@gmail.com> parents: 60202 diff changeset	214 good = ['/some/path']
3119c4d7276b Merged revisions 83209 via svnmerge from Senthil Kumaran <orsenthil@gmail.com> parents: 60202 diff changeset	215 bad = ['/some/path?name=value']
3119c4d7276b Merged revisions 83209 via svnmerge from Senthil Kumaran <orsenthil@gmail.com> parents: 60202 diff changeset	216
3119c4d7276b Merged revisions 83209 via svnmerge from Senthil Kumaran <orsenthil@gmail.com> parents: 60202 diff changeset	217 RobotTest(14, doc, good, bad)
3119c4d7276b Merged revisions 83209 via svnmerge from Senthil Kumaran <orsenthil@gmail.com> parents: 60202 diff changeset	218
63464 415ea2c3cfe8 Merged revisions 83238 via svnmerge from Georg Brandl <georg@python.org> parents: 63271 diff changeset	219 # 15. For issue #4108 (obey first * entry)
415ea2c3cfe8 Merged revisions 83238 via svnmerge from Georg Brandl <georg@python.org> parents: 63271 diff changeset	220 doc = """
415ea2c3cfe8 Merged revisions 83238 via svnmerge from Georg Brandl <georg@python.org> parents: 63271 diff changeset	221 User-agent: *
415ea2c3cfe8 Merged revisions 83238 via svnmerge from Georg Brandl <georg@python.org> parents: 63271 diff changeset	222 Disallow: /some/path
415ea2c3cfe8 Merged revisions 83238 via svnmerge from Georg Brandl <georg@python.org> parents: 63271 diff changeset	223
415ea2c3cfe8 Merged revisions 83238 via svnmerge from Georg Brandl <georg@python.org> parents: 63271 diff changeset	224 User-agent: *
415ea2c3cfe8 Merged revisions 83238 via svnmerge from Georg Brandl <georg@python.org> parents: 63271 diff changeset	225 Disallow: /another/path
415ea2c3cfe8 Merged revisions 83238 via svnmerge from Georg Brandl <georg@python.org> parents: 63271 diff changeset	226 """
415ea2c3cfe8 Merged revisions 83238 via svnmerge from Georg Brandl <georg@python.org> parents: 63271 diff changeset	227
415ea2c3cfe8 Merged revisions 83238 via svnmerge from Georg Brandl <georg@python.org> parents: 63271 diff changeset	228 good = ['/another/path']
415ea2c3cfe8 Merged revisions 83238 via svnmerge from Georg Brandl <georg@python.org> parents: 63271 diff changeset	229 bad = ['/some/path']
415ea2c3cfe8 Merged revisions 83238 via svnmerge from Georg Brandl <georg@python.org> parents: 63271 diff changeset	230
415ea2c3cfe8 Merged revisions 83238 via svnmerge from Georg Brandl <georg@python.org> parents: 63271 diff changeset	231 RobotTest(15, doc, good, bad)
415ea2c3cfe8 Merged revisions 83238 via svnmerge from Georg Brandl <georg@python.org> parents: 63271 diff changeset	232
48594 35b1a20470db Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48563 diff changeset	233
48535 dece17edbed0 Bug 3347: robotparser failed because it didn't convert bytes to string. Jeremy Hylton <jeremy@alum.mit.edu> parents: 48334 diff changeset	234 class NetworkTestCase(unittest.TestCase):
dece17edbed0 Bug 3347: robotparser failed because it didn't convert bytes to string. Jeremy Hylton <jeremy@alum.mit.edu> parents: 48334 diff changeset	235
dece17edbed0 Bug 3347: robotparser failed because it didn't convert bytes to string. Jeremy Hylton <jeremy@alum.mit.edu> parents: 48334 diff changeset	236 def testPasswordProtectedSite(self):
60202 5496503b8b1a Merged revision 79605 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 55593 diff changeset	237 support.requires('network')
64536 2400f55ea312 Merged revisions 84597-84599 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 63464 diff changeset	238 with support.transient_internet('mueblesmoraleda.com'):
2400f55ea312 Merged revisions 84597-84599 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 63464 diff changeset	239 url = 'http://mueblesmoraleda.com'
2400f55ea312 Merged revisions 84597-84599 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 63464 diff changeset	240 parser = urllib.robotparser.RobotFileParser()
2400f55ea312 Merged revisions 84597-84599 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 63464 diff changeset	241 parser.set_url(url)
2400f55ea312 Merged revisions 84597-84599 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 63464 diff changeset	242 try:
2400f55ea312 Merged revisions 84597-84599 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 63464 diff changeset	243 parser.read()
2400f55ea312 Merged revisions 84597-84599 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 63464 diff changeset	244 except URLError:
2400f55ea312 Merged revisions 84597-84599 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 63464 diff changeset	245 self.skipTest('%s is unavailable' % url)
2400f55ea312 Merged revisions 84597-84599 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 63464 diff changeset	246 self.assertEqual(parser.can_fetch("*", url+"/robots.txt"), False)
43095 4a07a288f5c4 Merged revisions 57620-57771 via svnmerge from Thomas Wouters <thomas@python.org> parents: 42501 diff changeset	247
48535 dece17edbed0 Bug 3347: robotparser failed because it didn't convert bytes to string. Jeremy Hylton <jeremy@alum.mit.edu> parents: 48334 diff changeset	248 def testPythonOrg(self):
60202 5496503b8b1a Merged revision 79605 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 55593 diff changeset	249 support.requires('network')
64536 2400f55ea312 Merged revisions 84597-84599 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 63464 diff changeset	250 with support.transient_internet('www.python.org'):
2400f55ea312 Merged revisions 84597-84599 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 63464 diff changeset	251 parser = urllib.robotparser.RobotFileParser(
2400f55ea312 Merged revisions 84597-84599 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 63464 diff changeset	252 "http://www.python.org/robots.txt")
2400f55ea312 Merged revisions 84597-84599 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 63464 diff changeset	253 parser.read()
2400f55ea312 Merged revisions 84597-84599 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 63464 diff changeset	254 self.assertTrue(
2400f55ea312 Merged revisions 84597-84599 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 63464 diff changeset	255 parser.can_fetch("*", "http://www.python.org/robots.txt"))
48535 dece17edbed0 Bug 3347: robotparser failed because it didn't convert bytes to string. Jeremy Hylton <jeremy@alum.mit.edu> parents: 48334 diff changeset	256
22088 736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	257 def test_main():
48535 dece17edbed0 Bug 3347: robotparser failed because it didn't convert bytes to string. Jeremy Hylton <jeremy@alum.mit.edu> parents: 48334 diff changeset	258 support.run_unittest(NetworkTestCase)
47201 9c930a32cebb #2621 rename test.test_support to test.support Benjamin Peterson <benjamin@python.org> parents: 43095 diff changeset	259 support.run_unittest(tests)
22088 736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	260
736fb5aec9b3 Correct various errors: Martin v. Löwis <martin@v.loewis.de> parents: diff changeset	261 if __name__=='__main__':
48563 b51cd5d3795c Merged revisions 65012,65035,65037-65040,65048,65057,65077,65091-65095,65097-65099,65127-65128,65131,65133-65136,65139,65149-65151,65155,65158-65159,65176-65178,65183-65184,65187-65190,65192,65194 via svnmerge from Georg Brandl <georg@python.org> parents: 48535 diff changeset	262 support.verbose = 1
41554 06145fbc7ab9 Merged revisions 53952-54987 via svnmerge from Guido van Rossum <guido@python.org> parents: 24364 diff changeset	263 test_main()

Mercurial > cpython-lru_cache-weakref

annotate Lib/test/test_robotparser.py @ 70817:dd775b1561ba