Mercurial > cpython-lru_cache-weakref
annotate Doc/lib/librobotparser.tex @ 16413:8819867f14ef legacy-trunk
Patch derived from Trent's 101162: a Python/C API testing framework.
STILL NEEDS UNIX BUILD CHANGES.
| author | Tim Peters <tim.peters@gmail.com> |
|---|---|
| date | Fri, 02 Feb 2001 05:57:15 +0000 |
| parents | c2bb5e4b04a8 |
| children | 346dcafc878b |
| rev | line source |
|---|---|
| 12198 | 1 \section{\module{robotparser} --- |
|
12530
c2bb5e4b04a8
\file and \filenq should not be used in section titles; they are not
Fred Drake <fdrake@acm.org>
parents:
12198
diff
changeset
|
2 Parser for robots.txt} |
| 12198 | 3 |
| 4 \declaremodule{standard}{robotparser} | |
| 5 \modulesynopsis{Accepts as input a list of lines or URL that refers to a | |
| 6 robots.txt file, parses the file, then builds a | |
| 7 set of rules from that list and answers questions | |
| 8 about fetchability of other URLs.} | |
| 9 \sectionauthor{Skip Montanaro}{skip@mojam.com} | |
| 10 | |
| 11 \index{WWW} | |
| 12 \index{World-Wide Web} | |
| 13 \index{URL} | |
| 14 \index{robots.txt} | |
| 15 | |
| 16 This module provides a single class, \class{RobotFileParser}, which answers | |
| 17 questions about whether or not a particular user agent can fetch a URL on | |
| 18 the web site that published the \file{robots.txt} file. For more details on | |
| 19 the structure of \file{robots.txt} files, see | |
| 20 \url{http://info.webcrawler.com/mak/projects/robots/norobots.html}. | |
| 21 | |
| 22 \begin{classdesc}{RobotFileParser}{} | |
| 23 | |
| 24 This class provides a set of methods to read, parse and answer questions | |
| 25 about a single \file{robots.txt} file. | |
| 26 | |
| 27 \begin{methoddesc}{set_url}{url} | |
| 28 Sets the URL referring to a \file{robots.txt} file. | |
| 29 \end{methoddesc} | |
| 30 | |
| 31 \begin{methoddesc}{read}{} | |
| 32 Reads the \file{robots.txt} URL and feeds it to the parser. | |
| 33 \end{methoddesc} | |
| 34 | |
| 35 \begin{methoddesc}{parse}{lines} | |
| 36 Parses the lines argument. | |
| 37 \end{methoddesc} | |
| 38 | |
| 39 \begin{methoddesc}{can_fetch}{useragent, url} | |
| 40 Returns true if the \var{useragent} is allowed to fetch the \var{url} | |
| 41 according to the rules contained in the parsed \file{robots.txt} file. | |
| 42 \end{methoddesc} | |
| 43 | |
| 44 \begin{methoddesc}{mtime}{} | |
| 45 Returns the time the \code{robots.txt} file was last fetched. This is | |
| 46 useful for long-running web spiders that need to check for new | |
| 47 \code{robots.txt} files periodically. | |
| 48 \end{methoddesc} | |
| 49 | |
| 50 \begin{methoddesc}{modified}{} | |
| 51 Sets the time the \code{robots.txt} file was last fetched to the current | |
| 52 time. | |
| 53 \end{methoddesc} | |
| 54 | |
| 55 \end{classdesc} | |
| 56 | |
| 57 The following example demonstrates basic use of the RobotFileParser class. | |
| 58 | |
| 59 \begin{verbatim} | |
| 60 >>> import robotparser | |
| 61 >>> rp = robotparser.RobotFileParser() | |
| 62 >>> rp.set_url("http://www.musi-cal.com/robots.txt") | |
| 63 >>> rp.read() | |
| 64 >>> rp.can_fetch("*", "http://www.musi-cal.com/cgi-bin/search?city=San+Francisco") | |
| 65 0 | |
| 66 >>> rp.can_fetch("*", "http://www.musi-cal.com/") | |
| 67 1 | |
| 68 \end{verbatim} |