cpython-lru_cache-weakref: Doc/lib/libtokenize.tex annotate

annotate Doc/lib/libtokenize.tex @ 39042:c02fa187e0f4 legacy-trunk

Part of SF patch #1484695. This removes dead code. The chksum was already verified in .frombuf() on the lines above. If there was a problem an exception is raised, so there was no way this condition could have been true.

author	Neal Norwitz <nnorwitz@gmail.com>
date	Mon, 10 Jul 2006 00:23:17 +0000
parents	8b6017c937ac
children

rev	line source
10238 86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	1 \section{\module{tokenize} ---
86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	2 Tokenizer for Python source}
86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	3
86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	4 \declaremodule{standard}{tokenize}
86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	5 \modulesynopsis{Lexical scanner for Python source code.}
86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	6 \moduleauthor{Ka Ping Yee}{}
86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	7 \sectionauthor{Fred L. Drake, Jr.}{fdrake@acm.org}
86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	8
86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	9
86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	10 The \module{tokenize} module provides a lexical scanner for Python
86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	11 source code, implemented in Python. The scanner in this module
86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	12 returns comments as tokens as well, making it useful for implementing
86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	13 ``pretty-printers,'' including colorizers for on-screen displays.
86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	14
18351 44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 17177 diff changeset	15 The primary entry point is a generator:
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 17177 diff changeset	16
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 17177 diff changeset	17 \begin{funcdesc}{generate_tokens}{readline}
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 17177 diff changeset	18 The \function{generate_tokens()} generator requires one argment,
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 17177 diff changeset	19 \var{readline}, which must be a callable object which
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 17177 diff changeset	20 provides the same interface as the \method{readline()} method of
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 17177 diff changeset	21 built-in file objects (see section~\ref{bltin-file-objects}). Each
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 17177 diff changeset	22 call to the function should return one line of input as a string.
10238 86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	23
18351 44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 17177 diff changeset	24 The generator produces 5-tuples with these members:
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 17177 diff changeset	25 the token type;
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 17177 diff changeset	26 the token string;
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 17177 diff changeset	27 a 2-tuple \code{(\var{srow}, \var{scol})} of ints specifying the
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 17177 diff changeset	28 row and column where the token begins in the source;
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 17177 diff changeset	29 a 2-tuple \code{(\var{erow}, \var{ecol})} of ints specifying the
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 17177 diff changeset	30 row and column where the token ends in the source;
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 17177 diff changeset	31 and the line on which the token was found.
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 17177 diff changeset	32 The line passed is the \emph{logical} line;
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 17177 diff changeset	33 continuation lines are included.
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 17177 diff changeset	34 \versionadded{2.2}
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 17177 diff changeset	35 \end{funcdesc}
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 17177 diff changeset	36
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 17177 diff changeset	37 An older entry point is retained for backward compatibility:
10238 86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	38
86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	39 \begin{funcdesc}{tokenize}{readline\optional{, tokeneater}}
86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	40 The \function{tokenize()} function accepts two parameters: one
18351 44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 17177 diff changeset	41 representing the input stream, and one providing an output mechanism
10238 86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	42 for \function{tokenize()}.
86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	43
86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	44 The first parameter, \var{readline}, must be a callable object which
10728 eea1ab028f43 Fix a couple of grammatical errors reported by Detlef Lannert Fred Drake <fdrake@acm.org> parents: 10238 diff changeset	45 provides the same interface as the \method{readline()} method of
10238 86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	46 built-in file objects (see section~\ref{bltin-file-objects}). Each
86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	47 call to the function should return one line of input as a string.
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	48 Alternately, \var{readline} may be a callable object that signals
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	49 completion by raising \exception{StopIteration}.
38170 8b6017c937ac Add \exception markup George Yoshida <dynkin@gmail.com> parents: 35063 diff changeset	50 \versionchanged[Added \exception{StopIteration} support]{2.5}
10238 86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	51
86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	52 The second parameter, \var{tokeneater}, must also be a callable
18351 44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 17177 diff changeset	53 object. It is called once for each token, with five arguments,
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 17177 diff changeset	54 corresponding to the tuples generated by \function{generate_tokens()}.
10238 86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	55 \end{funcdesc}
86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	56
86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	57
18351 44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 17177 diff changeset	58 All constants from the \refmodule{token} module are also exported from
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 17177 diff changeset	59 \module{tokenize}, as are two additional token type values that might be
10238 86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	60 passed to the \var{tokeneater} function by \function{tokenize()}:
86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	61
86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	62 \begin{datadesc}{COMMENT}
86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	63 Token value used to indicate a comment.
86642b7a7a60 Documentation for pyclbr and tokenize modules. Fred Drake <fdrake@acm.org> parents: diff changeset	64 \end{datadesc}
16854 bc89f116160b added description of NL token Skip Montanaro <skip@pobox.com> parents: 10728 diff changeset	65 \begin{datadesc}{NL}
17177 63cacdeb8045 Explain the difference between NL and NEWLINE. Ka-Ping Yee <ping@zesty.ca> parents: 16854 diff changeset	66 Token value used to indicate a non-terminating newline. The NEWLINE
63cacdeb8045 Explain the difference between NL and NEWLINE. Ka-Ping Yee <ping@zesty.ca> parents: 16854 diff changeset	67 token indicates the end of a logical line of Python code; NL tokens
63cacdeb8045 Explain the difference between NL and NEWLINE. Ka-Ping Yee <ping@zesty.ca> parents: 16854 diff changeset	68 are generated when a logical line of code is continued over multiple
63cacdeb8045 Explain the difference between NL and NEWLINE. Ka-Ping Yee <ping@zesty.ca> parents: 16854 diff changeset	69 physical lines.
16854 bc89f116160b added description of NL token Skip Montanaro <skip@pobox.com> parents: 10728 diff changeset	70 \end{datadesc}
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	71
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	72 Another function is provided to reverse the tokenization process.
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	73 This is useful for creating tools that tokenize a script, modify
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	74 the token stream, and write back the modified script.
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	75
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	76 \begin{funcdesc}{untokenize}{iterable}
35063 851458fffd34 fix typo. \var vs \variable Gregory P. Smith <greg@mad-scientist.com> parents: 35058 diff changeset	77 Converts tokens back into Python source code. The \var{iterable}
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	78 must return sequences with at least two elements, the token type and
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	79 the token string. Any additional sequence elements are ignored.
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	80
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	81 The reconstructed script is returned as a single string. The
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	82 result is guaranteed to tokenize back to match the input so that
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	83 the conversion is lossless and round-trips are assured. The
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	84 guarantee applies only to the token type and token string as
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	85 the spacing between tokens (column positions) may change.
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	86 \versionadded{2.5}
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	87 \end{funcdesc}
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	88
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	89 Example of a script re-writer that transforms float literals into
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	90 Decimal objects:
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	91 \begin{verbatim}
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	92 def decistmt(s):
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	93 """Substitute Decimals for floats in a string of statements.
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	94
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	95 >>> from decimal import Decimal
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	96 >>> s = 'print +21.3e-5*-.1234/81.7'
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	97 >>> decistmt(s)
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	98 "print +Decimal ('21.3e-5')*-Decimal ('.1234')/Decimal ('81.7')"
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	99
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	100 >>> exec(s)
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	101 -3.21716034272e-007
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	102 >>> exec(decistmt(s))
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	103 -3.217160342717258261933904529E-7
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	104
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	105 """
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	106 result = []
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	107 g = generate_tokens(StringIO(s).readline) # tokenize the string
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	108 for toknum, tokval, _, _, _ in g:
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	109 if toknum == NUMBER and '.' in tokval: # replace NUMBER tokens
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	110 result.extend([
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	111 (NAME, 'Decimal'),
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	112 (OP, '('),
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	113 (STRING, repr(tokval)),
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	114 (OP, ')')
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	115 ])
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	116 else:
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	117 result.append((toknum, tokval))
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	118 return untokenize(result)
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 18351 diff changeset	119 \end{verbatim}

Mercurial > cpython-lru_cache-weakref

annotate Doc/lib/libtokenize.tex @ 39042:c02fa187e0f4 legacy-trunk