Mercurial > cpython-lru_cache-weakref
annotate Lib/tokenize.py @ 80661:94de77bd0b4b 2.7
Issue #16194: document imp.load_dynamic problems
when read several extension modules from the same binary object file.
| author | Andrew Svetlov <andrew.svetlov@gmail.com> |
|---|---|
| date | Thu, 29 Nov 2012 19:51:28 +0200 |
| parents | eb7ea51e658e |
| children | 9b170d74a0a2 |
| rev | line source |
|---|---|
|
5173
ff4f8fa8a273
Fixed doc string, added __version__, fixed 1 bug.
Guido van Rossum <guido@python.org>
parents:
5172
diff
changeset
|
1 """Tokenization help for Python programs. |
| 558 | 2 |
|
18351
44a87c300f2b
Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents:
18148
diff
changeset
|
3 generate_tokens(readline) is a generator that breaks a stream of |
|
5266
1efc4273fdb7
Ka-Ping's muich improved version of March 26, 1997:
Guido van Rossum <guido@python.org>
parents:
5178
diff
changeset
|
4 text into Python tokens. It accepts a readline-like method which is called |
|
18351
44a87c300f2b
Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents:
18148
diff
changeset
|
5 repeatedly to get the next line of input (or "" for EOF). It generates |
|
44a87c300f2b
Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents:
18148
diff
changeset
|
6 5-tuples with these members: |
|
44a87c300f2b
Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents:
18148
diff
changeset
|
7 |
|
44a87c300f2b
Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents:
18148
diff
changeset
|
8 the token type (see token.py) |
|
44a87c300f2b
Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents:
18148
diff
changeset
|
9 the token (a string) |
|
44a87c300f2b
Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents:
18148
diff
changeset
|
10 the starting (row, column) indices of the token (a 2-tuple of ints) |
|
44a87c300f2b
Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents:
18148
diff
changeset
|
11 the ending (row, column) indices of the token (a 2-tuple of ints) |
|
44a87c300f2b
Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents:
18148
diff
changeset
|
12 the original line (string) |
|
44a87c300f2b
Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents:
18148
diff
changeset
|
13 |
|
44a87c300f2b
Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents:
18148
diff
changeset
|
14 It is designed to match the working of the Python tokenizer exactly, except |
|
44a87c300f2b
Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents:
18148
diff
changeset
|
15 that it produces COMMENT tokens for comments and gives type OP for all |
|
44a87c300f2b
Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents:
18148
diff
changeset
|
16 operators |
|
44a87c300f2b
Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents:
18148
diff
changeset
|
17 |
|
44a87c300f2b
Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents:
18148
diff
changeset
|
18 Older entry points |
|
44a87c300f2b
Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents:
18148
diff
changeset
|
19 tokenize_loop(readline, tokeneater) |
|
44a87c300f2b
Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents:
18148
diff
changeset
|
20 tokenize(readline, tokeneater=printtoken) |
|
44a87c300f2b
Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents:
18148
diff
changeset
|
21 are the same, except instead of generating tokens, tokeneater is a callback |
|
44a87c300f2b
Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents:
18148
diff
changeset
|
22 function to which the 5 fields described above are passed as 5 arguments, |
|
44a87c300f2b
Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents:
18148
diff
changeset
|
23 each time a new token is found.""" |
|
5173
ff4f8fa8a273
Fixed doc string, added __version__, fixed 1 bug.
Guido van Rossum <guido@python.org>
parents:
5172
diff
changeset
|
24 |
|
16899
320fa50a1a7d
Add __author__ and __credits__ variables.
Ka-Ping Yee <ping@zesty.ca>
parents:
16889
diff
changeset
|
25 __author__ = 'Ka-Ping Yee <ping@lfw.org>' |
| 56537 | 26 __credits__ = ('GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, ' |
| 27 'Skip Montanaro, Raymond Hettinger') | |
|
5173
ff4f8fa8a273
Fixed doc string, added __version__, fixed 1 bug.
Guido van Rossum <guido@python.org>
parents:
5172
diff
changeset
|
28 |
|
6531
e497a7fc0aa8
Redone (by Ka-Ping) using the new re module, and adding recognition
Guido van Rossum <guido@python.org>
parents:
5721
diff
changeset
|
29 import string, re |
| 5172 | 30 from token import * |
| 558 | 31 |
|
16889
0feb63490093
final round of __all__ lists (I hope) - skipped urllib2 because Moshe may be
Skip Montanaro <skip@pobox.com>
parents:
16514
diff
changeset
|
32 import token |
| 56537 | 33 __all__ = [x for x in dir(token) if not x.startswith("_")] |
| 34 __all__ += ["COMMENT", "tokenize", "generate_tokens", "NL", "untokenize"] | |
|
22585
93daa4fde087
Cleanup x so it is not left in module
Neal Norwitz <nnorwitz@gmail.com>
parents:
19621
diff
changeset
|
35 del x |
|
16889
0feb63490093
final round of __all__ lists (I hope) - skipped urllib2 because Moshe may be
Skip Montanaro <skip@pobox.com>
parents:
16514
diff
changeset
|
36 del token |
|
0feb63490093
final round of __all__ lists (I hope) - skipped urllib2 because Moshe may be
Skip Montanaro <skip@pobox.com>
parents:
16514
diff
changeset
|
37 |
|
5266
1efc4273fdb7
Ka-Ping's muich improved version of March 26, 1997:
Guido van Rossum <guido@python.org>
parents:
5178
diff
changeset
|
38 COMMENT = N_TOKENS |
|
1efc4273fdb7
Ka-Ping's muich improved version of March 26, 1997:
Guido van Rossum <guido@python.org>
parents:
5178
diff
changeset
|
39 tok_name[COMMENT] = 'COMMENT' |
|
7839
d4f5321b9d9e
Differentiate between NEWLINE token (an official newline) and NL token
Guido van Rossum <guido@python.org>
parents:
6532
diff
changeset
|
40 NL = N_TOKENS + 1 |
|
d4f5321b9d9e
Differentiate between NEWLINE token (an official newline) and NL token
Guido van Rossum <guido@python.org>
parents:
6532
diff
changeset
|
41 tok_name[NL] = 'NL' |
|
16889
0feb63490093
final round of __all__ lists (I hope) - skipped urllib2 because Moshe may be
Skip Montanaro <skip@pobox.com>
parents:
16514
diff
changeset
|
42 N_TOKENS += 2 |
|
5266
1efc4273fdb7
Ka-Ping's muich improved version of March 26, 1997:
Guido van Rossum <guido@python.org>
parents:
5178
diff
changeset
|
43 |
|
16514
90c600a819e9
String method conversion.
Eric S. Raymond <esr@thyrsus.com>
parents:
15898
diff
changeset
|
44 def group(*choices): return '(' + '|'.join(choices) + ')' |
|
28089
14f65d661084
Get rid of many apply() calls.
Guido van Rossum <guido@python.org>
parents:
26032
diff
changeset
|
45 def any(*choices): return group(*choices) + '*' |
|
14f65d661084
Get rid of many apply() calls.
Guido van Rossum <guido@python.org>
parents:
26032
diff
changeset
|
46 def maybe(*choices): return group(*choices) + '?' |
| 558 | 47 |
|
6531
e497a7fc0aa8
Redone (by Ka-Ping) using the new re module, and adding recognition
Guido van Rossum <guido@python.org>
parents:
5721
diff
changeset
|
48 Whitespace = r'[ \f\t]*' |
|
e497a7fc0aa8
Redone (by Ka-Ping) using the new re module, and adding recognition
Guido van Rossum <guido@python.org>
parents:
5721
diff
changeset
|
49 Comment = r'#[^\r\n]*' |
|
e497a7fc0aa8
Redone (by Ka-Ping) using the new re module, and adding recognition
Guido van Rossum <guido@python.org>
parents:
5721
diff
changeset
|
50 Ignore = Whitespace + any(r'\\\r?\n' + Whitespace) + maybe(Comment) |
|
e497a7fc0aa8
Redone (by Ka-Ping) using the new re module, and adding recognition
Guido van Rossum <guido@python.org>
parents:
5721
diff
changeset
|
51 Name = r'[a-zA-Z_]\w*' |
|
e497a7fc0aa8
Redone (by Ka-Ping) using the new re module, and adding recognition
Guido van Rossum <guido@python.org>
parents:
5721
diff
changeset
|
52 |
|
44780
61a11cc95632
Fix #1679: "0x" was taken as a valid integer literal.
Georg Brandl <georg@python.org>
parents:
44747
diff
changeset
|
53 Hexnumber = r'0[xX][\da-fA-F]+[lL]?' |
|
45801
b5cb7a678f38
Added PEP 3127 support to tokenize (with tests); added PEP 3127 to NEWS.
Eric Smith <eric@trueblade.com>
parents:
44780
diff
changeset
|
54 Octnumber = r'(0[oO][0-7]+)|(0[0-7]*)[lL]?' |
|
b5cb7a678f38
Added PEP 3127 support to tokenize (with tests); added PEP 3127 to NEWS.
Eric Smith <eric@trueblade.com>
parents:
44780
diff
changeset
|
55 Binnumber = r'0[bB][01]+[lL]?' |
|
6531
e497a7fc0aa8
Redone (by Ka-Ping) using the new re module, and adding recognition
Guido van Rossum <guido@python.org>
parents:
5721
diff
changeset
|
56 Decnumber = r'[1-9]\d*[lL]?' |
|
45801
b5cb7a678f38
Added PEP 3127 support to tokenize (with tests); added PEP 3127 to NEWS.
Eric Smith <eric@trueblade.com>
parents:
44780
diff
changeset
|
57 Intnumber = group(Hexnumber, Binnumber, Octnumber, Decnumber) |
|
6531
e497a7fc0aa8
Redone (by Ka-Ping) using the new re module, and adding recognition
Guido van Rossum <guido@python.org>
parents:
5721
diff
changeset
|
58 Exponent = r'[eE][-+]?\d+' |
|
e497a7fc0aa8
Redone (by Ka-Ping) using the new re module, and adding recognition
Guido van Rossum <guido@python.org>
parents:
5721
diff
changeset
|
59 Pointfloat = group(r'\d+\.\d*', r'\.\d+') + maybe(Exponent) |
|
19621
0bec699783d0
SF patch #455966: Allow leading 0 in float/imag literals.
Tim Peters <tim.peters@gmail.com>
parents:
19052
diff
changeset
|
60 Expfloat = r'\d+' + Exponent |
|
5266
1efc4273fdb7
Ka-Ping's muich improved version of March 26, 1997:
Guido van Rossum <guido@python.org>
parents:
5178
diff
changeset
|
61 Floatnumber = group(Pointfloat, Expfloat) |
|
19621
0bec699783d0
SF patch #455966: Allow leading 0 in float/imag literals.
Tim Peters <tim.peters@gmail.com>
parents:
19052
diff
changeset
|
62 Imagnumber = group(r'\d+[jJ]', Floatnumber + r'[jJ]') |
|
5266
1efc4273fdb7
Ka-Ping's muich improved version of March 26, 1997:
Guido van Rossum <guido@python.org>
parents:
5178
diff
changeset
|
63 Number = group(Imagnumber, Floatnumber, Intnumber) |
| 558 | 64 |
|
15080
0e0ea56e4be8
Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py).
Tim Peters <tim.peters@gmail.com>
parents:
14166
diff
changeset
|
65 # Tail end of ' string. |
|
0e0ea56e4be8
Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py).
Tim Peters <tim.peters@gmail.com>
parents:
14166
diff
changeset
|
66 Single = r"[^'\\]*(?:\\.[^'\\]*)*'" |
|
0e0ea56e4be8
Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py).
Tim Peters <tim.peters@gmail.com>
parents:
14166
diff
changeset
|
67 # Tail end of " string. |
|
0e0ea56e4be8
Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py).
Tim Peters <tim.peters@gmail.com>
parents:
14166
diff
changeset
|
68 Double = r'[^"\\]*(?:\\.[^"\\]*)*"' |
|
0e0ea56e4be8
Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py).
Tim Peters <tim.peters@gmail.com>
parents:
14166
diff
changeset
|
69 # Tail end of ''' string. |
|
0e0ea56e4be8
Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py).
Tim Peters <tim.peters@gmail.com>
parents:
14166
diff
changeset
|
70 Single3 = r"[^'\\]*(?:(?:\\.|'(?!''))[^'\\]*)*'''" |
|
0e0ea56e4be8
Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py).
Tim Peters <tim.peters@gmail.com>
parents:
14166
diff
changeset
|
71 # Tail end of """ string. |
|
0e0ea56e4be8
Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py).
Tim Peters <tim.peters@gmail.com>
parents:
14166
diff
changeset
|
72 Double3 = r'[^"\\]*(?:(?:\\.|"(?!""))[^"\\]*)*"""' |
|
77474
35d3a8ed7997
Issue #15054: Fix incorrect tokenization of 'b' and 'br' string literals.
Meador Inge <meadori@gmail.com>
parents:
68256
diff
changeset
|
73 Triple = group("[uUbB]?[rR]?'''", '[uUbB]?[rR]?"""') |
|
15080
0e0ea56e4be8
Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py).
Tim Peters <tim.peters@gmail.com>
parents:
14166
diff
changeset
|
74 # Single-line ' or " string. |
|
77474
35d3a8ed7997
Issue #15054: Fix incorrect tokenization of 'b' and 'br' string literals.
Meador Inge <meadori@gmail.com>
parents:
68256
diff
changeset
|
75 String = group(r"[uUbB]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*'", |
|
35d3a8ed7997
Issue #15054: Fix incorrect tokenization of 'b' and 'br' string literals.
Meador Inge <meadori@gmail.com>
parents:
68256
diff
changeset
|
76 r'[uUbB]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*"') |
| 558 | 77 |
|
15080
0e0ea56e4be8
Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py).
Tim Peters <tim.peters@gmail.com>
parents:
14166
diff
changeset
|
78 # Because of leftmost-then-longest match semantics, be sure to put the |
|
0e0ea56e4be8
Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py).
Tim Peters <tim.peters@gmail.com>
parents:
14166
diff
changeset
|
79 # longest operators first (e.g., if = came before ==, == would get |
|
0e0ea56e4be8
Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py).
Tim Peters <tim.peters@gmail.com>
parents:
14166
diff
changeset
|
80 # recognized as two instances of =). |
|
0e0ea56e4be8
Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py).
Tim Peters <tim.peters@gmail.com>
parents:
14166
diff
changeset
|
81 Operator = group(r"\*\*=?", r">>=?", r"<<=?", r"<>", r"!=", |
|
19052
3969fcbf9500
Add new tokens // and //=, in support of PEP 238.
Guido van Rossum <guido@python.org>
parents:
18781
diff
changeset
|
82 r"//=?", |
|
15080
0e0ea56e4be8
Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py).
Tim Peters <tim.peters@gmail.com>
parents:
14166
diff
changeset
|
83 r"[+\-*/%&|^=<>]=?", |
|
0e0ea56e4be8
Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py).
Tim Peters <tim.peters@gmail.com>
parents:
14166
diff
changeset
|
84 r"~") |
|
14166
e9c2f830f9ba
Update for augmented assignment, tested & approved by Guido.
Thomas Wouters <thomas@python.org>
parents:
14021
diff
changeset
|
85 |
| 558 | 86 Bracket = '[][(){}]' |
|
32888
c7e03cf8333e
PEP-0318, @decorator-style. In Guido's words:
Anthony Baxter <anthonybaxter@gmail.com>
parents:
28089
diff
changeset
|
87 Special = group(r'\r?\n', r'[:;.,`@]') |
| 5172 | 88 Funny = group(Operator, Bracket, Special) |
| 558 | 89 |
|
6531
e497a7fc0aa8
Redone (by Ka-Ping) using the new re module, and adding recognition
Guido van Rossum <guido@python.org>
parents:
5721
diff
changeset
|
90 PlainToken = group(Number, Funny, String, Name) |
| 5172 | 91 Token = Ignore + PlainToken |
| 558 | 92 |
|
15080
0e0ea56e4be8
Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py).
Tim Peters <tim.peters@gmail.com>
parents:
14166
diff
changeset
|
93 # First (or only) line of ' or " string. |
|
77474
35d3a8ed7997
Issue #15054: Fix incorrect tokenization of 'b' and 'br' string literals.
Meador Inge <meadori@gmail.com>
parents:
68256
diff
changeset
|
94 ContStr = group(r"[uUbB]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*" + |
|
15898
8366fcfb6090
Add tokenizer support and tests for u'', U"", uR'', Ur"", etc.
Ka-Ping Yee <ping@zesty.ca>
parents:
15873
diff
changeset
|
95 group("'", r'\\\r?\n'), |
|
77474
35d3a8ed7997
Issue #15054: Fix incorrect tokenization of 'b' and 'br' string literals.
Meador Inge <meadori@gmail.com>
parents:
68256
diff
changeset
|
96 r'[uUbB]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*' + |
|
15898
8366fcfb6090
Add tokenizer support and tests for u'', U"", uR'', Ur"", etc.
Ka-Ping Yee <ping@zesty.ca>
parents:
15873
diff
changeset
|
97 group('"', r'\\\r?\n')) |
|
80190
eb7ea51e658e
#16152: fix tokenize to ignore whitespace at the end of the code when no newline is found. Patch by Ned Batchelder.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
77474
diff
changeset
|
98 PseudoExtras = group(r'\\\r?\n|\Z', Comment, Triple) |
|
6531
e497a7fc0aa8
Redone (by Ka-Ping) using the new re module, and adding recognition
Guido van Rossum <guido@python.org>
parents:
5721
diff
changeset
|
99 PseudoToken = Whitespace + group(PseudoExtras, Number, Funny, ContStr, Name) |
|
5266
1efc4273fdb7
Ka-Ping's muich improved version of March 26, 1997:
Guido van Rossum <guido@python.org>
parents:
5178
diff
changeset
|
100 |
|
6531
e497a7fc0aa8
Redone (by Ka-Ping) using the new re module, and adding recognition
Guido van Rossum <guido@python.org>
parents:
5721
diff
changeset
|
101 tokenprog, pseudoprog, single3prog, double3prog = map( |
|
e497a7fc0aa8
Redone (by Ka-Ping) using the new re module, and adding recognition
Guido van Rossum <guido@python.org>
parents:
5721
diff
changeset
|
102 re.compile, (Token, PseudoToken, Single3, Double3)) |
|
6532
d369629e8c27
New, fixed version with proper r"..." and R"..." support from Ka-Ping.
Guido van Rossum <guido@python.org>
parents:
6531
diff
changeset
|
103 endprogs = {"'": re.compile(Single), '"': re.compile(Double), |
|
6531
e497a7fc0aa8
Redone (by Ka-Ping) using the new re module, and adding recognition
Guido van Rossum <guido@python.org>
parents:
5721
diff
changeset
|
104 "'''": single3prog, '"""': double3prog, |
|
6532
d369629e8c27
New, fixed version with proper r"..." and R"..." support from Ka-Ping.
Guido van Rossum <guido@python.org>
parents:
6531
diff
changeset
|
105 "r'''": single3prog, 'r"""': double3prog, |
|
15898
8366fcfb6090
Add tokenizer support and tests for u'', U"", uR'', Ur"", etc.
Ka-Ping Yee <ping@zesty.ca>
parents:
15873
diff
changeset
|
106 "u'''": single3prog, 'u"""': double3prog, |
|
8366fcfb6090
Add tokenizer support and tests for u'', U"", uR'', Ur"", etc.
Ka-Ping Yee <ping@zesty.ca>
parents:
15873
diff
changeset
|
107 "ur'''": single3prog, 'ur"""': double3prog, |
|
8366fcfb6090
Add tokenizer support and tests for u'', U"", uR'', Ur"", etc.
Ka-Ping Yee <ping@zesty.ca>
parents:
15873
diff
changeset
|
108 "R'''": single3prog, 'R"""': double3prog, |
|
8366fcfb6090
Add tokenizer support and tests for u'', U"", uR'', Ur"", etc.
Ka-Ping Yee <ping@zesty.ca>
parents:
15873
diff
changeset
|
109 "U'''": single3prog, 'U"""': double3prog, |
|
8366fcfb6090
Add tokenizer support and tests for u'', U"", uR'', Ur"", etc.
Ka-Ping Yee <ping@zesty.ca>
parents:
15873
diff
changeset
|
110 "uR'''": single3prog, 'uR"""': double3prog, |
|
8366fcfb6090
Add tokenizer support and tests for u'', U"", uR'', Ur"", etc.
Ka-Ping Yee <ping@zesty.ca>
parents:
15873
diff
changeset
|
111 "Ur'''": single3prog, 'Ur"""': double3prog, |
|
8366fcfb6090
Add tokenizer support and tests for u'', U"", uR'', Ur"", etc.
Ka-Ping Yee <ping@zesty.ca>
parents:
15873
diff
changeset
|
112 "UR'''": single3prog, 'UR"""': double3prog, |
|
44747
8da9ebf73026
Added bytes and b'' as aliases for str and ''
Christian Heimes <christian@cheimes.de>
parents:
40414
diff
changeset
|
113 "b'''": single3prog, 'b"""': double3prog, |
|
8da9ebf73026
Added bytes and b'' as aliases for str and ''
Christian Heimes <christian@cheimes.de>
parents:
40414
diff
changeset
|
114 "br'''": single3prog, 'br"""': double3prog, |
|
8da9ebf73026
Added bytes and b'' as aliases for str and ''
Christian Heimes <christian@cheimes.de>
parents:
40414
diff
changeset
|
115 "B'''": single3prog, 'B"""': double3prog, |
|
8da9ebf73026
Added bytes and b'' as aliases for str and ''
Christian Heimes <christian@cheimes.de>
parents:
40414
diff
changeset
|
116 "bR'''": single3prog, 'bR"""': double3prog, |
|
8da9ebf73026
Added bytes and b'' as aliases for str and ''
Christian Heimes <christian@cheimes.de>
parents:
40414
diff
changeset
|
117 "Br'''": single3prog, 'Br"""': double3prog, |
|
8da9ebf73026
Added bytes and b'' as aliases for str and ''
Christian Heimes <christian@cheimes.de>
parents:
40414
diff
changeset
|
118 "BR'''": single3prog, 'BR"""': double3prog, |
|
8da9ebf73026
Added bytes and b'' as aliases for str and ''
Christian Heimes <christian@cheimes.de>
parents:
40414
diff
changeset
|
119 'r': None, 'R': None, 'u': None, 'U': None, |
|
8da9ebf73026
Added bytes and b'' as aliases for str and ''
Christian Heimes <christian@cheimes.de>
parents:
40414
diff
changeset
|
120 'b': None, 'B': None} |
| 5172 | 121 |
|
25016
0ebec83eb3dd
Speed up the most egregious "if token in (long tuple)" cases by using
Guido van Rossum <guido@python.org>
parents:
23521
diff
changeset
|
122 triple_quoted = {} |
|
0ebec83eb3dd
Speed up the most egregious "if token in (long tuple)" cases by using
Guido van Rossum <guido@python.org>
parents:
23521
diff
changeset
|
123 for t in ("'''", '"""', |
|
0ebec83eb3dd
Speed up the most egregious "if token in (long tuple)" cases by using
Guido van Rossum <guido@python.org>
parents:
23521
diff
changeset
|
124 "r'''", 'r"""', "R'''", 'R"""', |
|
0ebec83eb3dd
Speed up the most egregious "if token in (long tuple)" cases by using
Guido van Rossum <guido@python.org>
parents:
23521
diff
changeset
|
125 "u'''", 'u"""', "U'''", 'U"""', |
|
0ebec83eb3dd
Speed up the most egregious "if token in (long tuple)" cases by using
Guido van Rossum <guido@python.org>
parents:
23521
diff
changeset
|
126 "ur'''", 'ur"""', "Ur'''", 'Ur"""', |
|
44747
8da9ebf73026
Added bytes and b'' as aliases for str and ''
Christian Heimes <christian@cheimes.de>
parents:
40414
diff
changeset
|
127 "uR'''", 'uR"""', "UR'''", 'UR"""', |
|
8da9ebf73026
Added bytes and b'' as aliases for str and ''
Christian Heimes <christian@cheimes.de>
parents:
40414
diff
changeset
|
128 "b'''", 'b"""', "B'''", 'B"""', |
|
8da9ebf73026
Added bytes and b'' as aliases for str and ''
Christian Heimes <christian@cheimes.de>
parents:
40414
diff
changeset
|
129 "br'''", 'br"""', "Br'''", 'Br"""', |
|
8da9ebf73026
Added bytes and b'' as aliases for str and ''
Christian Heimes <christian@cheimes.de>
parents:
40414
diff
changeset
|
130 "bR'''", 'bR"""', "BR'''", 'BR"""'): |
|
25016
0ebec83eb3dd
Speed up the most egregious "if token in (long tuple)" cases by using
Guido van Rossum <guido@python.org>
parents:
23521
diff
changeset
|
131 triple_quoted[t] = t |
|
0ebec83eb3dd
Speed up the most egregious "if token in (long tuple)" cases by using
Guido van Rossum <guido@python.org>
parents:
23521
diff
changeset
|
132 single_quoted = {} |
|
0ebec83eb3dd
Speed up the most egregious "if token in (long tuple)" cases by using
Guido van Rossum <guido@python.org>
parents:
23521
diff
changeset
|
133 for t in ("'", '"', |
|
0ebec83eb3dd
Speed up the most egregious "if token in (long tuple)" cases by using
Guido van Rossum <guido@python.org>
parents:
23521
diff
changeset
|
134 "r'", 'r"', "R'", 'R"', |
|
0ebec83eb3dd
Speed up the most egregious "if token in (long tuple)" cases by using
Guido van Rossum <guido@python.org>
parents:
23521
diff
changeset
|
135 "u'", 'u"', "U'", 'U"', |
|
0ebec83eb3dd
Speed up the most egregious "if token in (long tuple)" cases by using
Guido van Rossum <guido@python.org>
parents:
23521
diff
changeset
|
136 "ur'", 'ur"', "Ur'", 'Ur"', |
|
44747
8da9ebf73026
Added bytes and b'' as aliases for str and ''
Christian Heimes <christian@cheimes.de>
parents:
40414
diff
changeset
|
137 "uR'", 'uR"', "UR'", 'UR"', |
|
8da9ebf73026
Added bytes and b'' as aliases for str and ''
Christian Heimes <christian@cheimes.de>
parents:
40414
diff
changeset
|
138 "b'", 'b"', "B'", 'B"', |
|
8da9ebf73026
Added bytes and b'' as aliases for str and ''
Christian Heimes <christian@cheimes.de>
parents:
40414
diff
changeset
|
139 "br'", 'br"', "Br'", 'Br"', |
|
8da9ebf73026
Added bytes and b'' as aliases for str and ''
Christian Heimes <christian@cheimes.de>
parents:
40414
diff
changeset
|
140 "bR'", 'bR"', "BR'", 'BR"' ): |
|
25016
0ebec83eb3dd
Speed up the most egregious "if token in (long tuple)" cases by using
Guido van Rossum <guido@python.org>
parents:
23521
diff
changeset
|
141 single_quoted[t] = t |
|
0ebec83eb3dd
Speed up the most egregious "if token in (long tuple)" cases by using
Guido van Rossum <guido@python.org>
parents:
23521
diff
changeset
|
142 |
| 5172 | 143 tabsize = 8 |
|
14021
5b34b70ecf22
Convert some old-style string exceptions to class exceptions.
Fred Drake <fdrake@acm.org>
parents:
7839
diff
changeset
|
144 |
|
17178
707f95d05e69
Provide a StopTokenizing exception for conveniently exiting the loop.
Ka-Ping Yee <ping@zesty.ca>
parents:
16901
diff
changeset
|
145 class TokenError(Exception): pass |
|
707f95d05e69
Provide a StopTokenizing exception for conveniently exiting the loop.
Ka-Ping Yee <ping@zesty.ca>
parents:
16901
diff
changeset
|
146 |
|
707f95d05e69
Provide a StopTokenizing exception for conveniently exiting the loop.
Ka-Ping Yee <ping@zesty.ca>
parents:
16901
diff
changeset
|
147 class StopTokenizing(Exception): pass |
|
14021
5b34b70ecf22
Convert some old-style string exceptions to class exceptions.
Fred Drake <fdrake@acm.org>
parents:
7839
diff
changeset
|
148 |
|
48132
4b1ba50d3bdf
Remove a tuple unpacking in a parameter list to remove a SyntaxWarning raised
Brett Cannon <bcannon@gmail.com>
parents:
47480
diff
changeset
|
149 def printtoken(type, token, srow_scol, erow_ecol, line): # for testing |
|
4b1ba50d3bdf
Remove a tuple unpacking in a parameter list to remove a SyntaxWarning raised
Brett Cannon <bcannon@gmail.com>
parents:
47480
diff
changeset
|
150 srow, scol = srow_scol |
|
4b1ba50d3bdf
Remove a tuple unpacking in a parameter list to remove a SyntaxWarning raised
Brett Cannon <bcannon@gmail.com>
parents:
47480
diff
changeset
|
151 erow, ecol = erow_ecol |
|
5266
1efc4273fdb7
Ka-Ping's muich improved version of March 26, 1997:
Guido van Rossum <guido@python.org>
parents:
5178
diff
changeset
|
152 print "%d,%d-%d,%d:\t%s\t%s" % \ |
|
1efc4273fdb7
Ka-Ping's muich improved version of March 26, 1997:
Guido van Rossum <guido@python.org>
parents:
5178
diff
changeset
|
153 (srow, scol, erow, ecol, tok_name[type], repr(token)) |
| 5172 | 154 |
|
5266
1efc4273fdb7
Ka-Ping's muich improved version of March 26, 1997:
Guido van Rossum <guido@python.org>
parents:
5178
diff
changeset
|
155 def tokenize(readline, tokeneater=printtoken): |
|
23455
68c776ba5ea5
Added docstrings excerpted from Python Library Reference.
Raymond Hettinger <python@rcn.com>
parents:
22719
diff
changeset
|
156 """ |
|
68c776ba5ea5
Added docstrings excerpted from Python Library Reference.
Raymond Hettinger <python@rcn.com>
parents:
22719
diff
changeset
|
157 The tokenize() function accepts two parameters: one representing the |
|
68c776ba5ea5
Added docstrings excerpted from Python Library Reference.
Raymond Hettinger <python@rcn.com>
parents:
22719
diff
changeset
|
158 input stream, and one providing an output mechanism for tokenize(). |
|
23521
a90faa5f9417
Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents:
23455
diff
changeset
|
159 |
|
23455
68c776ba5ea5
Added docstrings excerpted from Python Library Reference.
Raymond Hettinger <python@rcn.com>
parents:
22719
diff
changeset
|
160 The first parameter, readline, must be a callable object which provides |
|
68c776ba5ea5
Added docstrings excerpted from Python Library Reference.
Raymond Hettinger <python@rcn.com>
parents:
22719
diff
changeset
|
161 the same interface as the readline() method of built-in file objects. |
|
23521
a90faa5f9417
Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents:
23455
diff
changeset
|
162 Each call to the function should return one line of input as a string. |
|
23455
68c776ba5ea5
Added docstrings excerpted from Python Library Reference.
Raymond Hettinger <python@rcn.com>
parents:
22719
diff
changeset
|
163 |
|
68c776ba5ea5
Added docstrings excerpted from Python Library Reference.
Raymond Hettinger <python@rcn.com>
parents:
22719
diff
changeset
|
164 The second parameter, tokeneater, must also be a callable object. It is |
|
68c776ba5ea5
Added docstrings excerpted from Python Library Reference.
Raymond Hettinger <python@rcn.com>
parents:
22719
diff
changeset
|
165 called once for each token, with five arguments, corresponding to the |
|
23521
a90faa5f9417
Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents:
23455
diff
changeset
|
166 tuples generated by generate_tokens(). |
|
23455
68c776ba5ea5
Added docstrings excerpted from Python Library Reference.
Raymond Hettinger <python@rcn.com>
parents:
22719
diff
changeset
|
167 """ |
|
17178
707f95d05e69
Provide a StopTokenizing exception for conveniently exiting the loop.
Ka-Ping Yee <ping@zesty.ca>
parents:
16901
diff
changeset
|
168 try: |
|
707f95d05e69
Provide a StopTokenizing exception for conveniently exiting the loop.
Ka-Ping Yee <ping@zesty.ca>
parents:
16901
diff
changeset
|
169 tokenize_loop(readline, tokeneater) |
|
707f95d05e69
Provide a StopTokenizing exception for conveniently exiting the loop.
Ka-Ping Yee <ping@zesty.ca>
parents:
16901
diff
changeset
|
170 except StopTokenizing: |
|
707f95d05e69
Provide a StopTokenizing exception for conveniently exiting the loop.
Ka-Ping Yee <ping@zesty.ca>
parents:
16901
diff
changeset
|
171 pass |
|
707f95d05e69
Provide a StopTokenizing exception for conveniently exiting the loop.
Ka-Ping Yee <ping@zesty.ca>
parents:
16901
diff
changeset
|
172 |
|
18351
44a87c300f2b
Turns out Neil didn't intend for *all* of his gen-branch work to get
Tim Peters <tim.peters@gmail.com>
parents:
18148
diff
changeset
|
173 # backwards compatible interface |
|
17178
707f95d05e69
Provide a StopTokenizing exception for conveniently exiting the loop.
Ka-Ping Yee <ping@zesty.ca>
parents:
16901
diff
changeset
|
174 def tokenize_loop(readline, tokeneater): |
|
18148
7bc473b2c654
Merging the gen-branch into the main line, at Guido's direction. Yay!
Tim Peters <tim.peters@gmail.com>
parents:
17178
diff
changeset
|
175 for token_info in generate_tokens(readline): |
|
28089
14f65d661084
Get rid of many apply() calls.
Guido van Rossum <guido@python.org>
parents:
26032
diff
changeset
|
176 tokeneater(*token_info) |
|
18148
7bc473b2c654
Merging the gen-branch into the main line, at Guido's direction. Yay!
Tim Peters <tim.peters@gmail.com>
parents:
17178
diff
changeset
|
177 |
|
39640
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
178 class Untokenizer: |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
179 |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
180 def __init__(self): |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
181 self.tokens = [] |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
182 self.prev_row = 1 |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
183 self.prev_col = 0 |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
184 |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
185 def add_whitespace(self, start): |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
186 row, col = start |
|
39641
f45451819491
Replace dead code with an assert.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39640
diff
changeset
|
187 assert row <= self.prev_row |
|
39640
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
188 col_offset = col - self.prev_col |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
189 if col_offset: |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
190 self.tokens.append(" " * col_offset) |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
191 |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
192 def untokenize(self, iterable): |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
193 for t in iterable: |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
194 if len(t) == 2: |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
195 self.compat(t, iterable) |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
196 break |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
197 tok_type, token, start, end, line = t |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
198 self.add_whitespace(start) |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
199 self.tokens.append(token) |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
200 self.prev_row, self.prev_col = end |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
201 if tok_type in (NEWLINE, NL): |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
202 self.prev_row += 1 |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
203 self.prev_col = 0 |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
204 return "".join(self.tokens) |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
205 |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
206 def compat(self, token, iterable): |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
207 startline = False |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
208 indents = [] |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
209 toks_append = self.tokens.append |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
210 toknum, tokval = token |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
211 if toknum in (NAME, NUMBER): |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
212 tokval += ' ' |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
213 if toknum in (NEWLINE, NL): |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
214 startline = True |
|
46142
31ba3fa1585b
Issue2495: tokenize.untokenize did not insert space between two consecutive string literals:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents:
45801
diff
changeset
|
215 prevstring = False |
|
39640
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
216 for tok in iterable: |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
217 toknum, tokval = tok[:2] |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
218 |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
219 if toknum in (NAME, NUMBER): |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
220 tokval += ' ' |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
221 |
|
46142
31ba3fa1585b
Issue2495: tokenize.untokenize did not insert space between two consecutive string literals:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents:
45801
diff
changeset
|
222 # Insert a space between two consecutive strings |
|
31ba3fa1585b
Issue2495: tokenize.untokenize did not insert space between two consecutive string literals:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents:
45801
diff
changeset
|
223 if toknum == STRING: |
|
31ba3fa1585b
Issue2495: tokenize.untokenize did not insert space between two consecutive string literals:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents:
45801
diff
changeset
|
224 if prevstring: |
|
31ba3fa1585b
Issue2495: tokenize.untokenize did not insert space between two consecutive string literals:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents:
45801
diff
changeset
|
225 tokval = ' ' + tokval |
|
31ba3fa1585b
Issue2495: tokenize.untokenize did not insert space between two consecutive string literals:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents:
45801
diff
changeset
|
226 prevstring = True |
|
31ba3fa1585b
Issue2495: tokenize.untokenize did not insert space between two consecutive string literals:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents:
45801
diff
changeset
|
227 else: |
|
31ba3fa1585b
Issue2495: tokenize.untokenize did not insert space between two consecutive string literals:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents:
45801
diff
changeset
|
228 prevstring = False |
|
31ba3fa1585b
Issue2495: tokenize.untokenize did not insert space between two consecutive string literals:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents:
45801
diff
changeset
|
229 |
|
39640
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
230 if toknum == INDENT: |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
231 indents.append(tokval) |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
232 continue |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
233 elif toknum == DEDENT: |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
234 indents.pop() |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
235 continue |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
236 elif toknum in (NEWLINE, NL): |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
237 startline = True |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
238 elif startline and indents: |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
239 toks_append(indents[-1]) |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
240 startline = False |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
241 toks_append(tokval) |
|
35058
8076424d455d
Add untokenize() function to allow full round-trip tokenization.
Raymond Hettinger <python@rcn.com>
parents:
32888
diff
changeset
|
242 |
|
8076424d455d
Add untokenize() function to allow full round-trip tokenization.
Raymond Hettinger <python@rcn.com>
parents:
32888
diff
changeset
|
243 def untokenize(iterable): |
|
8076424d455d
Add untokenize() function to allow full round-trip tokenization.
Raymond Hettinger <python@rcn.com>
parents:
32888
diff
changeset
|
244 """Transform tokens back into Python source code. |
|
8076424d455d
Add untokenize() function to allow full round-trip tokenization.
Raymond Hettinger <python@rcn.com>
parents:
32888
diff
changeset
|
245 |
|
8076424d455d
Add untokenize() function to allow full round-trip tokenization.
Raymond Hettinger <python@rcn.com>
parents:
32888
diff
changeset
|
246 Each element returned by the iterable must be a token sequence |
|
39640
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
247 with at least two elements, a token number and token value. If |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
248 only two tokens are passed, the resulting output is poor. |
|
35058
8076424d455d
Add untokenize() function to allow full round-trip tokenization.
Raymond Hettinger <python@rcn.com>
parents:
32888
diff
changeset
|
249 |
|
39640
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
250 Round-trip invariant for full input: |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
251 Untokenized source will match input source exactly |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
252 |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
253 Round-trip invariant for limited intput: |
|
35058
8076424d455d
Add untokenize() function to allow full round-trip tokenization.
Raymond Hettinger <python@rcn.com>
parents:
32888
diff
changeset
|
254 # Output text will tokenize the back to the input |
|
8076424d455d
Add untokenize() function to allow full round-trip tokenization.
Raymond Hettinger <python@rcn.com>
parents:
32888
diff
changeset
|
255 t1 = [tok[:2] for tok in generate_tokens(f.readline)] |
|
8076424d455d
Add untokenize() function to allow full round-trip tokenization.
Raymond Hettinger <python@rcn.com>
parents:
32888
diff
changeset
|
256 newcode = untokenize(t1) |
|
8076424d455d
Add untokenize() function to allow full round-trip tokenization.
Raymond Hettinger <python@rcn.com>
parents:
32888
diff
changeset
|
257 readline = iter(newcode.splitlines(1)).next |
|
46143
cb56330d7adc
Issue2495: tokenize.untokenize did not insert space between two consecutive string literals:
Amaury Forgeot d'Arc <amauryfa@gmail.com>
parents:
39495
diff
changeset
|
258 t2 = [tok[:2] for tok in generate_tokens(readline)] |
|
35058
8076424d455d
Add untokenize() function to allow full round-trip tokenization.
Raymond Hettinger <python@rcn.com>
parents:
32888
diff
changeset
|
259 assert t1 == t2 |
|
8076424d455d
Add untokenize() function to allow full round-trip tokenization.
Raymond Hettinger <python@rcn.com>
parents:
32888
diff
changeset
|
260 """ |
|
39640
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
261 ut = Untokenizer() |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
262 return ut.untokenize(iterable) |
|
35058
8076424d455d
Add untokenize() function to allow full round-trip tokenization.
Raymond Hettinger <python@rcn.com>
parents:
32888
diff
changeset
|
263 |
|
18148
7bc473b2c654
Merging the gen-branch into the main line, at Guido's direction. Yay!
Tim Peters <tim.peters@gmail.com>
parents:
17178
diff
changeset
|
264 def generate_tokens(readline): |
|
23455
68c776ba5ea5
Added docstrings excerpted from Python Library Reference.
Raymond Hettinger <python@rcn.com>
parents:
22719
diff
changeset
|
265 """ |
|
68c776ba5ea5
Added docstrings excerpted from Python Library Reference.
Raymond Hettinger <python@rcn.com>
parents:
22719
diff
changeset
|
266 The generate_tokens() generator requires one argment, readline, which |
|
68c776ba5ea5
Added docstrings excerpted from Python Library Reference.
Raymond Hettinger <python@rcn.com>
parents:
22719
diff
changeset
|
267 must be a callable object which provides the same interface as the |
|
68c776ba5ea5
Added docstrings excerpted from Python Library Reference.
Raymond Hettinger <python@rcn.com>
parents:
22719
diff
changeset
|
268 readline() method of built-in file objects. Each call to the function |
|
35058
8076424d455d
Add untokenize() function to allow full round-trip tokenization.
Raymond Hettinger <python@rcn.com>
parents:
32888
diff
changeset
|
269 should return one line of input as a string. Alternately, readline |
|
8076424d455d
Add untokenize() function to allow full round-trip tokenization.
Raymond Hettinger <python@rcn.com>
parents:
32888
diff
changeset
|
270 can be a callable function terminating with StopIteration: |
|
8076424d455d
Add untokenize() function to allow full round-trip tokenization.
Raymond Hettinger <python@rcn.com>
parents:
32888
diff
changeset
|
271 readline = open(myfile).next # Example of alternate readline |
|
23521
a90faa5f9417
Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents:
23455
diff
changeset
|
272 |
|
23455
68c776ba5ea5
Added docstrings excerpted from Python Library Reference.
Raymond Hettinger <python@rcn.com>
parents:
22719
diff
changeset
|
273 The generator produces 5-tuples with these members: the token type; the |
|
68c776ba5ea5
Added docstrings excerpted from Python Library Reference.
Raymond Hettinger <python@rcn.com>
parents:
22719
diff
changeset
|
274 token string; a 2-tuple (srow, scol) of ints specifying the row and |
|
68c776ba5ea5
Added docstrings excerpted from Python Library Reference.
Raymond Hettinger <python@rcn.com>
parents:
22719
diff
changeset
|
275 column where the token begins in the source; a 2-tuple (erow, ecol) of |
|
68c776ba5ea5
Added docstrings excerpted from Python Library Reference.
Raymond Hettinger <python@rcn.com>
parents:
22719
diff
changeset
|
276 ints specifying the row and column where the token ends in the source; |
|
68c776ba5ea5
Added docstrings excerpted from Python Library Reference.
Raymond Hettinger <python@rcn.com>
parents:
22719
diff
changeset
|
277 and the line on which the token was found. The line passed is the |
|
23521
a90faa5f9417
Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents:
23455
diff
changeset
|
278 logical line; continuation lines are included. |
|
23455
68c776ba5ea5
Added docstrings excerpted from Python Library Reference.
Raymond Hettinger <python@rcn.com>
parents:
22719
diff
changeset
|
279 """ |
|
5266
1efc4273fdb7
Ka-Ping's muich improved version of March 26, 1997:
Guido van Rossum <guido@python.org>
parents:
5178
diff
changeset
|
280 lnum = parenlev = continued = 0 |
|
18781
01ae962df885
Use string.ascii_letters instead of string.letters (SF bug #226706).
Fred Drake <fdrake@acm.org>
parents:
18609
diff
changeset
|
281 namechars, numchars = string.ascii_letters + '_', '0123456789' |
|
5277
7df5a0d1f523
Ping's latest. Fixes triple quoted strings ending in odd
Guido van Rossum <guido@python.org>
parents:
5266
diff
changeset
|
282 contstr, needcont = '', 0 |
|
7839
d4f5321b9d9e
Differentiate between NEWLINE token (an official newline) and NL token
Guido van Rossum <guido@python.org>
parents:
6532
diff
changeset
|
283 contline = None |
| 5172 | 284 indents = [0] |
|
5266
1efc4273fdb7
Ka-Ping's muich improved version of March 26, 1997:
Guido van Rossum <guido@python.org>
parents:
5178
diff
changeset
|
285 |
| 5172 | 286 while 1: # loop over lines in stream |
|
35058
8076424d455d
Add untokenize() function to allow full round-trip tokenization.
Raymond Hettinger <python@rcn.com>
parents:
32888
diff
changeset
|
287 try: |
|
8076424d455d
Add untokenize() function to allow full round-trip tokenization.
Raymond Hettinger <python@rcn.com>
parents:
32888
diff
changeset
|
288 line = readline() |
|
8076424d455d
Add untokenize() function to allow full round-trip tokenization.
Raymond Hettinger <python@rcn.com>
parents:
32888
diff
changeset
|
289 except StopIteration: |
|
8076424d455d
Add untokenize() function to allow full round-trip tokenization.
Raymond Hettinger <python@rcn.com>
parents:
32888
diff
changeset
|
290 line = '' |
| 56537 | 291 lnum += 1 |
| 5172 | 292 pos, max = 0, len(line) |
| 558 | 293 |
| 5172 | 294 if contstr: # continued string |
|
5277
7df5a0d1f523
Ping's latest. Fixes triple quoted strings ending in odd
Guido van Rossum <guido@python.org>
parents:
5266
diff
changeset
|
295 if not line: |
|
7df5a0d1f523
Ping's latest. Fixes triple quoted strings ending in odd
Guido van Rossum <guido@python.org>
parents:
5266
diff
changeset
|
296 raise TokenError, ("EOF in multi-line string", strstart) |
|
6531
e497a7fc0aa8
Redone (by Ka-Ping) using the new re module, and adding recognition
Guido van Rossum <guido@python.org>
parents:
5721
diff
changeset
|
297 endmatch = endprog.match(line) |
|
e497a7fc0aa8
Redone (by Ka-Ping) using the new re module, and adding recognition
Guido van Rossum <guido@python.org>
parents:
5721
diff
changeset
|
298 if endmatch: |
|
e497a7fc0aa8
Redone (by Ka-Ping) using the new re module, and adding recognition
Guido van Rossum <guido@python.org>
parents:
5721
diff
changeset
|
299 pos = end = endmatch.end(0) |
|
18148
7bc473b2c654
Merging the gen-branch into the main line, at Guido's direction. Yay!
Tim Peters <tim.peters@gmail.com>
parents:
17178
diff
changeset
|
300 yield (STRING, contstr + line[:end], |
|
39640
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
301 strstart, (lnum, end), contline + line) |
|
5277
7df5a0d1f523
Ping's latest. Fixes triple quoted strings ending in odd
Guido van Rossum <guido@python.org>
parents:
5266
diff
changeset
|
302 contstr, needcont = '', 0 |
|
7839
d4f5321b9d9e
Differentiate between NEWLINE token (an official newline) and NL token
Guido van Rossum <guido@python.org>
parents:
6532
diff
changeset
|
303 contline = None |
|
5277
7df5a0d1f523
Ping's latest. Fixes triple quoted strings ending in odd
Guido van Rossum <guido@python.org>
parents:
5266
diff
changeset
|
304 elif needcont and line[-2:] != '\\\n' and line[-3:] != '\\\r\n': |
|
18148
7bc473b2c654
Merging the gen-branch into the main line, at Guido's direction. Yay!
Tim Peters <tim.peters@gmail.com>
parents:
17178
diff
changeset
|
305 yield (ERRORTOKEN, contstr + line, |
|
7839
d4f5321b9d9e
Differentiate between NEWLINE token (an official newline) and NL token
Guido van Rossum <guido@python.org>
parents:
6532
diff
changeset
|
306 strstart, (lnum, len(line)), contline) |
| 5172 | 307 contstr = '' |
|
7839
d4f5321b9d9e
Differentiate between NEWLINE token (an official newline) and NL token
Guido van Rossum <guido@python.org>
parents:
6532
diff
changeset
|
308 contline = None |
|
5277
7df5a0d1f523
Ping's latest. Fixes triple quoted strings ending in odd
Guido van Rossum <guido@python.org>
parents:
5266
diff
changeset
|
309 continue |
| 5172 | 310 else: |
| 311 contstr = contstr + line | |
|
7839
d4f5321b9d9e
Differentiate between NEWLINE token (an official newline) and NL token
Guido van Rossum <guido@python.org>
parents:
6532
diff
changeset
|
312 contline = contline + line |
| 5172 | 313 continue |
| 314 | |
|
5266
1efc4273fdb7
Ka-Ping's muich improved version of March 26, 1997:
Guido van Rossum <guido@python.org>
parents:
5178
diff
changeset
|
315 elif parenlev == 0 and not continued: # new statement |
| 5172 | 316 if not line: break |
| 317 column = 0 | |
|
5266
1efc4273fdb7
Ka-Ping's muich improved version of March 26, 1997:
Guido van Rossum <guido@python.org>
parents:
5178
diff
changeset
|
318 while pos < max: # measure leading whitespace |
|
56534
fa6188c47b63
pep8ify if blocks
Benjamin Peterson <benjamin@python.org>
parents:
48132
diff
changeset
|
319 if line[pos] == ' ': |
| 56537 | 320 column += 1 |
|
56534
fa6188c47b63
pep8ify if blocks
Benjamin Peterson <benjamin@python.org>
parents:
48132
diff
changeset
|
321 elif line[pos] == '\t': |
|
56535
45b62725377a
use floor division and add a test that exercises the tabsize codepath
Benjamin Peterson <benjamin@python.org>
parents:
56534
diff
changeset
|
322 column = (column//tabsize + 1)*tabsize |
|
56534
fa6188c47b63
pep8ify if blocks
Benjamin Peterson <benjamin@python.org>
parents:
48132
diff
changeset
|
323 elif line[pos] == '\f': |
|
fa6188c47b63
pep8ify if blocks
Benjamin Peterson <benjamin@python.org>
parents:
48132
diff
changeset
|
324 column = 0 |
|
fa6188c47b63
pep8ify if blocks
Benjamin Peterson <benjamin@python.org>
parents:
48132
diff
changeset
|
325 else: |
|
fa6188c47b63
pep8ify if blocks
Benjamin Peterson <benjamin@python.org>
parents:
48132
diff
changeset
|
326 break |
| 56537 | 327 pos += 1 |
|
56534
fa6188c47b63
pep8ify if blocks
Benjamin Peterson <benjamin@python.org>
parents:
48132
diff
changeset
|
328 if pos == max: |
|
fa6188c47b63
pep8ify if blocks
Benjamin Peterson <benjamin@python.org>
parents:
48132
diff
changeset
|
329 break |
|
5266
1efc4273fdb7
Ka-Ping's muich improved version of March 26, 1997:
Guido van Rossum <guido@python.org>
parents:
5178
diff
changeset
|
330 |
|
1efc4273fdb7
Ka-Ping's muich improved version of March 26, 1997:
Guido van Rossum <guido@python.org>
parents:
5178
diff
changeset
|
331 if line[pos] in '#\r\n': # skip comments or blank lines |
|
39640
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
332 if line[pos] == '#': |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
333 comment_token = line[pos:].rstrip('\r\n') |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
334 nl_pos = pos + len(comment_token) |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
335 yield (COMMENT, comment_token, |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
336 (lnum, pos), (lnum, pos + len(comment_token)), line) |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
337 yield (NL, line[nl_pos:], |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
338 (lnum, nl_pos), (lnum, len(line)), line) |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
339 else: |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
340 yield ((NL, COMMENT)[line[pos] == '#'], line[pos:], |
|
5266
1efc4273fdb7
Ka-Ping's muich improved version of March 26, 1997:
Guido van Rossum <guido@python.org>
parents:
5178
diff
changeset
|
341 (lnum, pos), (lnum, len(line)), line) |
|
1efc4273fdb7
Ka-Ping's muich improved version of March 26, 1997:
Guido van Rossum <guido@python.org>
parents:
5178
diff
changeset
|
342 continue |
| 558 | 343 |
| 5172 | 344 if column > indents[-1]: # count indents or dedents |
| 345 indents.append(column) | |
|
18148
7bc473b2c654
Merging the gen-branch into the main line, at Guido's direction. Yay!
Tim Peters <tim.peters@gmail.com>
parents:
17178
diff
changeset
|
346 yield (INDENT, line[:pos], (lnum, 0), (lnum, pos), line) |
| 5172 | 347 while column < indents[-1]: |
|
35119
04868959f1fe
SF bug #1224621: tokenize module does not detect inconsistent dedents
Raymond Hettinger <python@rcn.com>
parents:
35058
diff
changeset
|
348 if column not in indents: |
|
04868959f1fe
SF bug #1224621: tokenize module does not detect inconsistent dedents
Raymond Hettinger <python@rcn.com>
parents:
35058
diff
changeset
|
349 raise IndentationError( |
|
39495
14ec9cfd9760
Make tabnanny recognize IndentationErrors raised by tokenize.
Georg Brandl <georg@python.org>
parents:
37314
diff
changeset
|
350 "unindent does not match any outer indentation level", |
|
14ec9cfd9760
Make tabnanny recognize IndentationErrors raised by tokenize.
Georg Brandl <georg@python.org>
parents:
37314
diff
changeset
|
351 ("<tokenize>", lnum, pos, line)) |
| 5172 | 352 indents = indents[:-1] |
|
18148
7bc473b2c654
Merging the gen-branch into the main line, at Guido's direction. Yay!
Tim Peters <tim.peters@gmail.com>
parents:
17178
diff
changeset
|
353 yield (DEDENT, '', (lnum, pos), (lnum, pos), line) |
| 5172 | 354 |
| 355 else: # continued statement | |
|
5277
7df5a0d1f523
Ping's latest. Fixes triple quoted strings ending in odd
Guido van Rossum <guido@python.org>
parents:
5266
diff
changeset
|
356 if not line: |
|
7df5a0d1f523
Ping's latest. Fixes triple quoted strings ending in odd
Guido van Rossum <guido@python.org>
parents:
5266
diff
changeset
|
357 raise TokenError, ("EOF in multi-line statement", (lnum, 0)) |
| 5172 | 358 continued = 0 |
| 359 | |
| 360 while pos < max: | |
|
6531
e497a7fc0aa8
Redone (by Ka-Ping) using the new re module, and adding recognition
Guido van Rossum <guido@python.org>
parents:
5721
diff
changeset
|
361 pseudomatch = pseudoprog.match(line, pos) |
|
e497a7fc0aa8
Redone (by Ka-Ping) using the new re module, and adding recognition
Guido van Rossum <guido@python.org>
parents:
5721
diff
changeset
|
362 if pseudomatch: # scan for tokens |
|
e497a7fc0aa8
Redone (by Ka-Ping) using the new re module, and adding recognition
Guido van Rossum <guido@python.org>
parents:
5721
diff
changeset
|
363 start, end = pseudomatch.span(1) |
|
5277
7df5a0d1f523
Ping's latest. Fixes triple quoted strings ending in odd
Guido van Rossum <guido@python.org>
parents:
5266
diff
changeset
|
364 spos, epos, pos = (lnum, start), (lnum, end), end |
|
80190
eb7ea51e658e
#16152: fix tokenize to ignore whitespace at the end of the code when no newline is found. Patch by Ned Batchelder.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
77474
diff
changeset
|
365 if start == end: |
|
eb7ea51e658e
#16152: fix tokenize to ignore whitespace at the end of the code when no newline is found. Patch by Ned Batchelder.
Ezio Melotti <ezio.melotti@gmail.com>
parents:
77474
diff
changeset
|
366 continue |
|
5266
1efc4273fdb7
Ka-Ping's muich improved version of March 26, 1997:
Guido van Rossum <guido@python.org>
parents:
5178
diff
changeset
|
367 token, initial = line[start:end], line[start] |
| 5172 | 368 |
|
17178
707f95d05e69
Provide a StopTokenizing exception for conveniently exiting the loop.
Ka-Ping Yee <ping@zesty.ca>
parents:
16901
diff
changeset
|
369 if initial in numchars or \ |
|
707f95d05e69
Provide a StopTokenizing exception for conveniently exiting the loop.
Ka-Ping Yee <ping@zesty.ca>
parents:
16901
diff
changeset
|
370 (initial == '.' and token != '.'): # ordinary number |
|
18148
7bc473b2c654
Merging the gen-branch into the main line, at Guido's direction. Yay!
Tim Peters <tim.peters@gmail.com>
parents:
17178
diff
changeset
|
371 yield (NUMBER, token, spos, epos, line) |
|
5266
1efc4273fdb7
Ka-Ping's muich improved version of March 26, 1997:
Guido van Rossum <guido@python.org>
parents:
5178
diff
changeset
|
372 elif initial in '\r\n': |
|
39640
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
373 yield (NL if parenlev > 0 else NEWLINE, |
|
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
374 token, spos, epos, line) |
|
5266
1efc4273fdb7
Ka-Ping's muich improved version of March 26, 1997:
Guido van Rossum <guido@python.org>
parents:
5178
diff
changeset
|
375 elif initial == '#': |
|
39640
5cce028874bc
Bug fixes large and small for tokenize.
Jeremy Hylton <jeremy@alum.mit.edu>
parents:
39495
diff
changeset
|
376 assert not token.endswith("\n") |
|
18148
7bc473b2c654
Merging the gen-branch into the main line, at Guido's direction. Yay!
Tim Peters <tim.peters@gmail.com>
parents:
17178
diff
changeset
|
377 yield (COMMENT, token, spos, epos, line) |
|
25016
0ebec83eb3dd
Speed up the most egregious "if token in (long tuple)" cases by using
Guido van Rossum <guido@python.org>
parents:
23521
diff
changeset
|
378 elif token in triple_quoted: |
| 5172 | 379 endprog = endprogs[token] |
|
6531
e497a7fc0aa8
Redone (by Ka-Ping) using the new re module, and adding recognition
Guido van Rossum <guido@python.org>
parents:
5721
diff
changeset
|
380 endmatch = endprog.match(line, pos) |
|
e497a7fc0aa8
Redone (by Ka-Ping) using the new re module, and adding recognition
Guido van Rossum <guido@python.org>
parents:
5721
diff
changeset
|
381 if endmatch: # all on one line |
|
e497a7fc0aa8
Redone (by Ka-Ping) using the new re module, and adding recognition
Guido van Rossum <guido@python.org>
parents:
5721
diff
changeset
|
382 pos = endmatch.end(0) |
|
5266
1efc4273fdb7
Ka-Ping's muich improved version of March 26, 1997:
Guido van Rossum <guido@python.org>
parents:
5178
diff
changeset
|
383 token = line[start:pos] |
|
18148
7bc473b2c654
Merging the gen-branch into the main line, at Guido's direction. Yay!
Tim Peters <tim.peters@gmail.com>
parents:
17178
diff
changeset
|
384 yield (STRING, token, spos, (lnum, pos), line) |
| 5172 | 385 else: |
|
5266
1efc4273fdb7
Ka-Ping's muich improved version of March 26, 1997:
Guido van Rossum <guido@python.org>
parents:
5178
diff
changeset
|
386 strstart = (lnum, start) # multiple lines |
|
1efc4273fdb7
Ka-Ping's muich improved version of March 26, 1997:
Guido van Rossum <guido@python.org>
parents:
5178
diff
changeset
|
387 contstr = line[start:] |
|
7839
d4f5321b9d9e
Differentiate between NEWLINE token (an official newline) and NL token
Guido van Rossum <guido@python.org>
parents:
6532
diff
changeset
|
388 contline = line |
| 5172 | 389 break |
|
25016
0ebec83eb3dd
Speed up the most egregious "if token in (long tuple)" cases by using
Guido van Rossum <guido@python.org>
parents:
23521
diff
changeset
|
390 elif initial in single_quoted or \ |
|
0ebec83eb3dd
Speed up the most egregious "if token in (long tuple)" cases by using
Guido van Rossum <guido@python.org>
parents:
23521
diff
changeset
|
391 token[:2] in single_quoted or \ |
|
0ebec83eb3dd
Speed up the most egregious "if token in (long tuple)" cases by using
Guido van Rossum <guido@python.org>
parents:
23521
diff
changeset
|
392 token[:3] in single_quoted: |
| 5172 | 393 if token[-1] == '\n': # continued string |
|
5266
1efc4273fdb7
Ka-Ping's muich improved version of March 26, 1997:
Guido van Rossum <guido@python.org>
parents:
5178
diff
changeset
|
394 strstart = (lnum, start) |
|
15898
8366fcfb6090
Add tokenizer support and tests for u'', U"", uR'', Ur"", etc.
Ka-Ping Yee <ping@zesty.ca>
parents:
15873
diff
changeset
|
395 endprog = (endprogs[initial] or endprogs[token[1]] or |
|
8366fcfb6090
Add tokenizer support and tests for u'', U"", uR'', Ur"", etc.
Ka-Ping Yee <ping@zesty.ca>
parents:
15873
diff
changeset
|
396 endprogs[token[2]]) |
|
5277
7df5a0d1f523
Ping's latest. Fixes triple quoted strings ending in odd
Guido van Rossum <guido@python.org>
parents:
5266
diff
changeset
|
397 contstr, needcont = line[start:], 1 |
|
7839
d4f5321b9d9e
Differentiate between NEWLINE token (an official newline) and NL token
Guido van Rossum <guido@python.org>
parents:
6532
diff
changeset
|
398 contline = line |
| 5172 | 399 break |
| 400 else: # ordinary string | |
|
18148
7bc473b2c654
Merging the gen-branch into the main line, at Guido's direction. Yay!
Tim Peters <tim.peters@gmail.com>
parents:
17178
diff
changeset
|
401 yield (STRING, token, spos, epos, line) |
|
6531
e497a7fc0aa8
Redone (by Ka-Ping) using the new re module, and adding recognition
Guido van Rossum <guido@python.org>
parents:
5721
diff
changeset
|
402 elif initial in namechars: # ordinary name |
|
18148
7bc473b2c654
Merging the gen-branch into the main line, at Guido's direction. Yay!
Tim Peters <tim.peters@gmail.com>
parents:
17178
diff
changeset
|
403 yield (NAME, token, spos, epos, line) |
|
6531
e497a7fc0aa8
Redone (by Ka-Ping) using the new re module, and adding recognition
Guido van Rossum <guido@python.org>
parents:
5721
diff
changeset
|
404 elif initial == '\\': # continued stmt |
|
e497a7fc0aa8
Redone (by Ka-Ping) using the new re module, and adding recognition
Guido van Rossum <guido@python.org>
parents:
5721
diff
changeset
|
405 continued = 1 |
| 5172 | 406 else: |
|
56534
fa6188c47b63
pep8ify if blocks
Benjamin Peterson <benjamin@python.org>
parents:
48132
diff
changeset
|
407 if initial in '([{': |
| 56537 | 408 parenlev += 1 |
|
56534
fa6188c47b63
pep8ify if blocks
Benjamin Peterson <benjamin@python.org>
parents:
48132
diff
changeset
|
409 elif initial in ')]}': |
| 56537 | 410 parenlev -= 1 |
|
18148
7bc473b2c654
Merging the gen-branch into the main line, at Guido's direction. Yay!
Tim Peters <tim.peters@gmail.com>
parents:
17178
diff
changeset
|
411 yield (OP, token, spos, epos, line) |
| 5172 | 412 else: |
|
18148
7bc473b2c654
Merging the gen-branch into the main line, at Guido's direction. Yay!
Tim Peters <tim.peters@gmail.com>
parents:
17178
diff
changeset
|
413 yield (ERRORTOKEN, line[pos], |
|
5277
7df5a0d1f523
Ping's latest. Fixes triple quoted strings ending in odd
Guido van Rossum <guido@python.org>
parents:
5266
diff
changeset
|
414 (lnum, pos), (lnum, pos+1), line) |
| 56537 | 415 pos += 1 |
| 5172 | 416 |
| 417 for indent in indents[1:]: # pop remaining indent levels | |
|
18148
7bc473b2c654
Merging the gen-branch into the main line, at Guido's direction. Yay!
Tim Peters <tim.peters@gmail.com>
parents:
17178
diff
changeset
|
418 yield (DEDENT, '', (lnum, 0), (lnum, 0), '') |
|
7bc473b2c654
Merging the gen-branch into the main line, at Guido's direction. Yay!
Tim Peters <tim.peters@gmail.com>
parents:
17178
diff
changeset
|
419 yield (ENDMARKER, '', (lnum, 0), (lnum, 0), '') |
| 5172 | 420 |
| 421 if __name__ == '__main__': # testing | |
| 422 import sys | |
|
56534
fa6188c47b63
pep8ify if blocks
Benjamin Peterson <benjamin@python.org>
parents:
48132
diff
changeset
|
423 if len(sys.argv) > 1: |
|
fa6188c47b63
pep8ify if blocks
Benjamin Peterson <benjamin@python.org>
parents:
48132
diff
changeset
|
424 tokenize(open(sys.argv[1]).readline) |
|
fa6188c47b63
pep8ify if blocks
Benjamin Peterson <benjamin@python.org>
parents:
48132
diff
changeset
|
425 else: |
|
fa6188c47b63
pep8ify if blocks
Benjamin Peterson <benjamin@python.org>
parents:
48132
diff
changeset
|
426 tokenize(sys.stdin.readline) |