Mercurial > cpython-lru_cache-weakref

annotate Doc/lib/libtokenize.tex @ 25838:8540887bcec6 2.2

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Backport of 2.183: Fix memory leak in add_subclass() found by NealN with valgrind.
author Guido van Rossum <guido@python.org>
date Fri, 18 Oct 2002 13:42:21 +0000
parents 44a87c300f2b
children 8076424d455d
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
10238
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
1 \section{\module{tokenize} ---
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
2 Tokenizer for Python source}
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
3
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
4 \declaremodule{standard}{tokenize}
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
5 \modulesynopsis{Lexical scanner for Python source code.}
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
6 \moduleauthor{Ka Ping Yee}{}
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
7 \sectionauthor{Fred L. Drake, Jr.}{fdrake@acm.org}
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
8
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
9
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
10 The \module{tokenize} module provides a lexical scanner for Python
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
11 source code, implemented in Python. The scanner in this module
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
12 returns comments as tokens as well, making it useful for implementing
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
13 ``pretty-printers,'' including colorizers for on-screen displays.
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
14
18351
44a87c300f2b Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents: 17177
diff changeset
15 The primary entry point is a generator:
44a87c300f2b Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents: 17177
diff changeset
16
44a87c300f2b Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents: 17177
diff changeset
17 \begin{funcdesc}{generate_tokens}{readline}
44a87c300f2b Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents: 17177
diff changeset
18 The \function{generate_tokens()} generator requires one argment,
44a87c300f2b Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents: 17177
diff changeset
19 \var{readline}, which must be a callable object which
44a87c300f2b Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents: 17177
diff changeset
20 provides the same interface as the \method{readline()} method of
44a87c300f2b Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents: 17177
diff changeset
21 built-in file objects (see section~\ref{bltin-file-objects}). Each
44a87c300f2b Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents: 17177
diff changeset
22 call to the function should return one line of input as a string.
10238
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
23
18351
44a87c300f2b Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents: 17177
diff changeset
24 The generator produces 5-tuples with these members:
44a87c300f2b Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents: 17177
diff changeset
25 the token type;
44a87c300f2b Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents: 17177
diff changeset
26 the token string;
44a87c300f2b Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents: 17177
diff changeset
27 a 2-tuple \code{(\var{srow}, \var{scol})} of ints specifying the
44a87c300f2b Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents: 17177
diff changeset
28 row and column where the token begins in the source;
44a87c300f2b Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents: 17177
diff changeset
29 a 2-tuple \code{(\var{erow}, \var{ecol})} of ints specifying the
44a87c300f2b Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents: 17177
diff changeset
30 row and column where the token ends in the source;
44a87c300f2b Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents: 17177
diff changeset
31 and the line on which the token was found.
44a87c300f2b Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents: 17177
diff changeset
32 The line passed is the \emph{logical} line;
44a87c300f2b Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents: 17177
diff changeset
33 continuation lines are included.
44a87c300f2b Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents: 17177
diff changeset
34 \versionadded{2.2}
44a87c300f2b Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents: 17177
diff changeset
35 \end{funcdesc}
44a87c300f2b Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents: 17177
diff changeset
36
44a87c300f2b Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents: 17177
diff changeset
37 An older entry point is retained for backward compatibility:
10238
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
38
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
39 \begin{funcdesc}{tokenize}{readline\optional{, tokeneater}}
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
40 The \function{tokenize()} function accepts two parameters: one
18351
44a87c300f2b Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents: 17177
diff changeset
41 representing the input stream, and one providing an output mechanism
10238
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
42 for \function{tokenize()}.
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
43
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
44 The first parameter, \var{readline}, must be a callable object which
10728
eea1ab028f43 Fix a couple of grammatical errors reported by Detlef Lannert
Fred Drake <fdrake@acm.org>
parents: 10238
diff changeset
45 provides the same interface as the \method{readline()} method of
10238
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
46 built-in file objects (see section~\ref{bltin-file-objects}). Each
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
47 call to the function should return one line of input as a string.
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
48
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
49 The second parameter, \var{tokeneater}, must also be a callable
18351
44a87c300f2b Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents: 17177
diff changeset
50 object. It is called once for each token, with five arguments,
44a87c300f2b Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents: 17177
diff changeset
51 corresponding to the tuples generated by \function{generate_tokens()}.
10238
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
52 \end{funcdesc}
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
53
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
54
18351
44a87c300f2b Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents: 17177
diff changeset
55 All constants from the \refmodule{token} module are also exported from
44a87c300f2b Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents: 17177
diff changeset
56 \module{tokenize}, as are two additional token type values that might be
10238
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
57 passed to the \var{tokeneater} function by \function{tokenize()}:
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
58
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
59 \begin{datadesc}{COMMENT}
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
60 Token value used to indicate a comment.
86642b7a7a60 Documentation for pyclbr and tokenize modules.
Fred Drake <fdrake@acm.org>
parents:
diff changeset
61 \end{datadesc}
16854
bc89f116160b added description of NL token
Skip Montanaro <skip@pobox.com>
parents: 10728
diff changeset
62 \begin{datadesc}{NL}
17177
63cacdeb8045 Explain the difference between NL and NEWLINE.
Ka-Ping Yee <ping@zesty.ca>
parents: 16854
diff changeset
63 Token value used to indicate a non-terminating newline. The NEWLINE
63cacdeb8045 Explain the difference between NL and NEWLINE.
Ka-Ping Yee <ping@zesty.ca>
parents: 16854
diff changeset
64 token indicates the end of a logical line of Python code; NL tokens
63cacdeb8045 Explain the difference between NL and NEWLINE.
Ka-Ping Yee <ping@zesty.ca>
parents: 16854
diff changeset
65 are generated when a logical line of code is continued over multiple
63cacdeb8045 Explain the difference between NL and NEWLINE.
Ka-Ping Yee <ping@zesty.ca>
parents: 16854
diff changeset
66 physical lines.
16854
bc89f116160b added description of NL token
Skip Montanaro <skip@pobox.com>
parents: 10728
diff changeset
67 \end{datadesc}