Mercurial > cpython-lru_cache-weakref

annotate Doc/library/tokenize.rst @ 54294:dba767bc3155 legacy-trunk

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
remove mention of old ctypes version
author Benjamin Peterson <benjamin@python.org>
date Sat, 23 May 2009 20:59:09 +0000
parents 24df3434ccf7
children 8838c56b6952
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
42604
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
1
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
2 :mod:`tokenize` --- Tokenizer for Python source
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
3 ===============================================
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
4
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
5 .. module:: tokenize
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
6 :synopsis: Lexical scanner for Python source code.
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
7 .. moduleauthor:: Ka Ping Yee
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
8 .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org>
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
9
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
10
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
11 The :mod:`tokenize` module provides a lexical scanner for Python source code,
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
12 implemented in Python. The scanner in this module returns comments as tokens as
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
13 well, making it useful for implementing "pretty-printers," including colorizers
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
14 for on-screen displays.
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
15
43629
4f662ba09f1f Add :term: for generators.
Georg Brandl <georg@python.org>
parents: 42604
diff changeset
16 The primary entry point is a :term:`generator`:
42604
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
17
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
18 .. function:: generate_tokens(readline)
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
19
47519
1c03090b2a0a #3028: tokenize passes the physical line.
Georg Brandl <georg@python.org>
parents: 45429
diff changeset
20 The :func:`generate_tokens` generator requires one argument, *readline*,
1c03090b2a0a #3028: tokenize passes the physical line.
Georg Brandl <georg@python.org>
parents: 45429
diff changeset
21 which must be a callable object which provides the same interface as the
42604
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
22 :meth:`readline` method of built-in file objects (see section
47519
1c03090b2a0a #3028: tokenize passes the physical line.
Georg Brandl <georg@python.org>
parents: 45429
diff changeset
23 :ref:`bltin-file-objects`). Each call to the function should return one line
1c03090b2a0a #3028: tokenize passes the physical line.
Georg Brandl <georg@python.org>
parents: 45429
diff changeset
24 of input as a string.
42604
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
25
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
26 The generator produces 5-tuples with these members: the token type; the token
47519
1c03090b2a0a #3028: tokenize passes the physical line.
Georg Brandl <georg@python.org>
parents: 45429
diff changeset
27 string; a 2-tuple ``(srow, scol)`` of ints specifying the row and column
1c03090b2a0a #3028: tokenize passes the physical line.
Georg Brandl <georg@python.org>
parents: 45429
diff changeset
28 where the token begins in the source; a 2-tuple ``(erow, ecol)`` of ints
1c03090b2a0a #3028: tokenize passes the physical line.
Georg Brandl <georg@python.org>
parents: 45429
diff changeset
29 specifying the row and column where the token ends in the source; and the
47520
24df3434ccf7 Argh, I read it wrong. Reverted 64036 and added a clarifying remark.
Georg Brandl <georg@python.org>
parents: 47519
diff changeset
30 line on which the token was found. The line passed (the last tuple item) is
24df3434ccf7 Argh, I read it wrong. Reverted 64036 and added a clarifying remark.
Georg Brandl <georg@python.org>
parents: 47519
diff changeset
31 the *logical* line; continuation lines are included.
42604
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
32
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
33 .. versionadded:: 2.2
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
34
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
35 An older entry point is retained for backward compatibility:
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
36
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
37
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
38 .. function:: tokenize(readline[, tokeneater])
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
39
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
40 The :func:`tokenize` function accepts two parameters: one representing the input
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
41 stream, and one providing an output mechanism for :func:`tokenize`.
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
42
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
43 The first parameter, *readline*, must be a callable object which provides the
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
44 same interface as the :meth:`readline` method of built-in file objects (see
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
45 section :ref:`bltin-file-objects`). Each call to the function should return one
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
46 line of input as a string. Alternately, *readline* may be a callable object that
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
47 signals completion by raising :exc:`StopIteration`.
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
48
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
49 .. versionchanged:: 2.5
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
50 Added :exc:`StopIteration` support.
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
51
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
52 The second parameter, *tokeneater*, must also be a callable object. It is
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
53 called once for each token, with five arguments, corresponding to the tuples
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
54 generated by :func:`generate_tokens`.
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
55
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
56 All constants from the :mod:`token` module are also exported from
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
57 :mod:`tokenize`, as are two additional token type values that might be passed to
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
58 the *tokeneater* function by :func:`tokenize`:
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
59
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
60
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
61 .. data:: COMMENT
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
62
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
63 Token value used to indicate a comment.
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
64
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
65
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
66 .. data:: NL
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
67
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
68 Token value used to indicate a non-terminating newline. The NEWLINE token
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
69 indicates the end of a logical line of Python code; NL tokens are generated when
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
70 a logical line of code is continued over multiple physical lines.
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
71
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
72 Another function is provided to reverse the tokenization process. This is useful
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
73 for creating tools that tokenize a script, modify the token stream, and write
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
74 back the modified script.
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
75
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
76
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
77 .. function:: untokenize(iterable)
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
78
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
79 Converts tokens back into Python source code. The *iterable* must return
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
80 sequences with at least two elements, the token type and the token string. Any
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
81 additional sequence elements are ignored.
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
82
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
83 The reconstructed script is returned as a single string. The result is
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
84 guaranteed to tokenize back to match the input so that the conversion is
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
85 lossless and round-trips are assured. The guarantee applies only to the token
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
86 type and token string as the spacing between tokens (column positions) may
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
87 change.
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
88
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
89 .. versionadded:: 2.5
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
90
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
91 Example of a script re-writer that transforms float literals into Decimal
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
92 objects::
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
93
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
94 def decistmt(s):
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
95 """Substitute Decimals for floats in a string of statements.
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
96
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
97 >>> from decimal import Decimal
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
98 >>> s = 'print +21.3e-5*-.1234/81.7'
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
99 >>> decistmt(s)
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
100 "print +Decimal ('21.3e-5')*-Decimal ('.1234')/Decimal ('81.7')"
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
101
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
102 >>> exec(s)
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
103 -3.21716034272e-007
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
104 >>> exec(decistmt(s))
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
105 -3.217160342717258261933904529E-7
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
106
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
107 """
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
108 result = []
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
109 g = generate_tokens(StringIO(s).readline) # tokenize the string
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
110 for toknum, tokval, _, _, _ in g:
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
111 if toknum == NUMBER and '.' in tokval: # replace NUMBER tokens
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
112 result.extend([
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
113 (NAME, 'Decimal'),
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
114 (OP, '('),
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
115 (STRING, repr(tokval)),
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
116 (OP, ')')
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
117 ])
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
118 else:
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
119 result.append((toknum, tokval))
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
120 return untokenize(result)
9e1529bf0442 Move the 3k reST doc tree in place.
Georg Brandl <georg@python.org>
parents:
diff changeset
121