Mercurial > cpython-lru_cache-weakref

annotate Doc/library/robotparser.rst @ 79675:c0a2294de787 2.7

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Issue #16199: Backport 2to3 typo fix from 3.2.
author Chris Jerdonek <chris.jerdonek@gmail.com>
date Thu, 11 Oct 2012 16:15:46 -0700
parents 8aab13719c02
children 274e68e2a5a6
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
42604
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
1
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
2 :mod:`robotparser` --- Parser for robots.txt
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
3 =============================================
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
4
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
5 .. module:: robotparser
46590
a517eccc783b Wrap some long lines.
Skip Montanaro <skip@pobox.com>
parents: 46556
diff changeset
6 :synopsis: Loads a robots.txt file and answers questions about
46593
fef6d6ba78a4 Fix markup.
Georg Brandl <georg@python.org>
parents: 46590
diff changeset
7 fetchability of other URLs.
44230
6b517ac10e8e Merged revisions 59407-59422 via svnmerge from
Christian Heimes <christian@cheimes.de>
parents: 42604
diff changeset
8 .. sectionauthor:: Skip Montanaro <skip@pobox.com>
42604
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
9
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
10
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
11 .. index::
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
12 single: WWW
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
13 single: World Wide Web
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
14 single: URL
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
15 single: robots.txt
50522
ee9ced92843a Merged revisions 68133-68134,68141-68142,68145-68146,68148-68149,68159-68162,68166,68171-68174,68179,68195-68196,68210,68214-68215,68217-68222 via svnmerge from
Georg Brandl <georg@python.org>
parents: 47929
diff changeset
16
47929
c03b223c2b69 Doc that robotparse has been renamed urllib.robotparser in Python 3.0.
Brett Cannon <bcannon@gmail.com>
parents: 46593
diff changeset
17 .. note::
c03b223c2b69 Doc that robotparse has been renamed urllib.robotparser in Python 3.0.
Brett Cannon <bcannon@gmail.com>
parents: 46593
diff changeset
18 The :mod:`robotparser` module has been renamed :mod:`urllib.robotparser` in
76732
8aab13719c02 Use Python 3 instead of 3.0.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 50522
diff changeset
19 Python 3.
47929
c03b223c2b69 Doc that robotparse has been renamed urllib.robotparser in Python 3.0.
Brett Cannon <bcannon@gmail.com>
parents: 46593
diff changeset
20 The :term:`2to3` tool will automatically adapt imports when converting
76732
8aab13719c02 Use Python 3 instead of 3.0.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 50522
diff changeset
21 your sources to Python 3.
42604
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
22
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
23 This module provides a single class, :class:`RobotFileParser`, which answers
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
24 questions about whether or not a particular user agent can fetch a URL on the
45764
c07c5c7a914e Fix lots of broken links in the docs, found by Sphinx' external link checker.
Georg Brandl <georg@python.org>
parents: 44230
diff changeset
25 Web site that published the :file:`robots.txt` file. For more details on the
c07c5c7a914e Fix lots of broken links in the docs, found by Sphinx' external link checker.
Georg Brandl <georg@python.org>
parents: 44230
diff changeset
26 structure of :file:`robots.txt` files, see http://www.robotstxt.org/orig.html.
42604
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
27
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
28
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
29 .. class:: RobotFileParser()
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
30
46590
a517eccc783b Wrap some long lines.
Skip Montanaro <skip@pobox.com>
parents: 46556
diff changeset
31 This class provides a set of methods to read, parse and answer questions
a517eccc783b Wrap some long lines.
Skip Montanaro <skip@pobox.com>
parents: 46556
diff changeset
32 about a single :file:`robots.txt` file.
42604
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
33
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
34
46556
8b718fc15f3d Merged revisions 62490 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 45764
diff changeset
35 .. method:: set_url(url)
42604
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
36
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
37 Sets the URL referring to a :file:`robots.txt` file.
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
38
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
39
46556
8b718fc15f3d Merged revisions 62490 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 45764
diff changeset
40 .. method:: read()
42604
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
41
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
42 Reads the :file:`robots.txt` URL and feeds it to the parser.
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
43
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
44
46556
8b718fc15f3d Merged revisions 62490 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 45764
diff changeset
45 .. method:: parse(lines)
42604
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
46
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
47 Parses the lines argument.
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
48
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
49
46556
8b718fc15f3d Merged revisions 62490 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 45764
diff changeset
50 .. method:: can_fetch(useragent, url)
42604
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
51
46590
a517eccc783b Wrap some long lines.
Skip Montanaro <skip@pobox.com>
parents: 46556
diff changeset
52 Returns ``True`` if the *useragent* is allowed to fetch the *url*
a517eccc783b Wrap some long lines.
Skip Montanaro <skip@pobox.com>
parents: 46556
diff changeset
53 according to the rules contained in the parsed :file:`robots.txt`
a517eccc783b Wrap some long lines.
Skip Montanaro <skip@pobox.com>
parents: 46556
diff changeset
54 file.
42604
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
55
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
56
46556
8b718fc15f3d Merged revisions 62490 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 45764
diff changeset
57 .. method:: mtime()
42604
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
58
46590
a517eccc783b Wrap some long lines.
Skip Montanaro <skip@pobox.com>
parents: 46556
diff changeset
59 Returns the time the ``robots.txt`` file was last fetched. This is
a517eccc783b Wrap some long lines.
Skip Montanaro <skip@pobox.com>
parents: 46556
diff changeset
60 useful for long-running web spiders that need to check for new
a517eccc783b Wrap some long lines.
Skip Montanaro <skip@pobox.com>
parents: 46556
diff changeset
61 ``robots.txt`` files periodically.
42604
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
62
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
63
46556
8b718fc15f3d Merged revisions 62490 via svnmerge from
Benjamin Peterson <benjamin@python.org>
parents: 45764
diff changeset
64 .. method:: modified()
42604
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
65
46590
a517eccc783b Wrap some long lines.
Skip Montanaro <skip@pobox.com>
parents: 46556
diff changeset
66 Sets the time the ``robots.txt`` file was last fetched to the current
a517eccc783b Wrap some long lines.
Skip Montanaro <skip@pobox.com>
parents: 46556
diff changeset
67 time.
42604
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
68
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
69 The following example demonstrates basic use of the RobotFileParser class. ::
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
70
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
71 >>> import robotparser
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
72 >>> rp = robotparser.RobotFileParser()
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
73 >>> rp.set_url("http://www.musi-cal.com/robots.txt")
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
74 >>> rp.read()
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
75 >>> rp.can_fetch("*", "http://www.musi-cal.com/cgi-bin/search?city=San+Francisco")
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
76 False
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
77 >>> rp.can_fetch("*", "http://www.musi-cal.com/")
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
78 True
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
79