cpython-lru_cache-weakref: Lib/tokenize.py annotate

annotate Lib/tokenize.py @ 74135:9c19df6c8ea0 3.2

News entry for Issue12798

author	Senthil Kumaran <senthil@uthcode.com>
date	Fri, 23 Dec 2011 11:11:21 +0800
parents	3c9232a684ae
children	b07488490001 85254bb6c1c6

rev	line source
5173 ff4f8fa8a273 Fixed doc string, added __version__, fixed 1 bug. Guido van Rossum <guido@python.org> parents: 5172 diff changeset	1 """Tokenization help for Python programs.
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	2
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	3 tokenize(readline) is a generator that breaks a stream of bytes into
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	4 Python tokens. It decodes the bytes according to PEP-0263 for
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	5 determining source file encoding.
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	6
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	7 It accepts a readline-like method which is called repeatedly to get the
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	8 next line of input (or b"" for EOF). It generates 5-tuples with these
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	9 members:
18351 44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	10
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	11 the token type (see token.py)
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	12 the token (a string)
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	13 the starting (row, column) indices of the token (a 2-tuple of ints)
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	14 the ending (row, column) indices of the token (a 2-tuple of ints)
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	15 the original line (string)
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	16
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	17 It is designed to match the working of the Python tokenizer exactly, except
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	18 that it produces COMMENT tokens for comments and gives type OP for all
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	19 operators. Additionally, all token lists start with an ENCODING token
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	20 which tells you which encoding was used to decode the bytes stream.
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	21 """
5173 ff4f8fa8a273 Fixed doc string, added __version__, fixed 1 bug. Guido van Rossum <guido@python.org> parents: 5172 diff changeset	22
16899 320fa50a1a7d Add __author__ and __credits__ variables. Ka-Ping Yee <ping@zesty.ca> parents: 16889 diff changeset	23 __author__ = 'Ka-Ping Yee <ping@lfw.org>'
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	24 __credits__ = ('GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, '
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	25 'Skip Montanaro, Raymond Hettinger, Trent Nelson, '
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	26 'Michael Foord')
68096 ea260d393cde Issue #11074: Make 'tokenize' so it can be reloaded. Brett Cannon <bcannon@gmail.com> parents: 66148 diff changeset	27 import builtins
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	28 import re
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	29 import sys
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	30 from token import *
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	31 from codecs import lookup, BOM_UTF8
64581 fea86c5e3385 Experiment: Let collections.namedtuple() do the work. This should work now that _collections is pre-built. The buildbots will tell us shortly. Raymond Hettinger <python@rcn.com> parents: 64577 diff changeset	32 import collections
66097 306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	33 from io import TextIOWrapper
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	34 cookie_re = re.compile("coding[:=]\s*([-\w.]+)")
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	35
16889 0feb63490093 final round of __all__ lists (I hope) - skipped urllib2 because Moshe may be Skip Montanaro <skip@pobox.com> parents: 16514 diff changeset	36 import token
66148 98bd3ca1ed86 Issue #10386: Added __all__ to token module; this simplifies importing Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66097 diff changeset	37 __all__ = token.__all__ + ["COMMENT", "tokenize", "detect_encoding",
98bd3ca1ed86 Issue #10386: Added __all__ to token module; this simplifies importing Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66097 diff changeset	38 "NL", "untokenize", "ENCODING", "TokenInfo"]
16889 0feb63490093 final round of __all__ lists (I hope) - skipped urllib2 because Moshe may be Skip Montanaro <skip@pobox.com> parents: 16514 diff changeset	39 del token
0feb63490093 final round of __all__ lists (I hope) - skipped urllib2 because Moshe may be Skip Montanaro <skip@pobox.com> parents: 16514 diff changeset	40
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	41 COMMENT = N_TOKENS
1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	42 tok_name[COMMENT] = 'COMMENT'
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	43 NL = N_TOKENS + 1
d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	44 tok_name[NL] = 'NL'
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	45 ENCODING = N_TOKENS + 2
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	46 tok_name[ENCODING] = 'ENCODING'
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	47 N_TOKENS += 3
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	48
64581 fea86c5e3385 Experiment: Let collections.namedtuple() do the work. This should work now that _collections is pre-built. The buildbots will tell us shortly. Raymond Hettinger <python@rcn.com> parents: 64577 diff changeset	49 class TokenInfo(collections.namedtuple('TokenInfo', 'type string start end line')):
53597 5f81c45bdf6d Remove dependency on the collections module. Raymond Hettinger <python@rcn.com> parents: 53584 diff changeset	50 def __repr__(self):
64582 71b264e1bb71 A little bit more readable repr method. Raymond Hettinger <python@rcn.com> parents: 64581 diff changeset	51 annotated_type = '%d (%s)' % (self.type, tok_name[self.type])
71b264e1bb71 A little bit more readable repr method. Raymond Hettinger <python@rcn.com> parents: 64581 diff changeset	52 return ('TokenInfo(type=%s, string=%r, start=%r, end=%r, line=%r)' %
71b264e1bb71 A little bit more readable repr method. Raymond Hettinger <python@rcn.com> parents: 64581 diff changeset	53 self._replace(type=annotated_type))
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	54
16514 90c600a819e9 String method conversion. Eric S. Raymond <esr@thyrsus.com> parents: 15898 diff changeset	55 def group(*choices): return '(' + '\|'.join(choices) + ')'
28089 14f65d661084 Get rid of many apply() calls. Guido van Rossum <guido@python.org> parents: 26032 diff changeset	56 def any(choices): return group(choices) + '*'
14f65d661084 Get rid of many apply() calls. Guido van Rossum <guido@python.org> parents: 26032 diff changeset	57 def maybe(choices): return group(choices) + '?'
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	58
48769 83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	59 # Note: we use unicode matching for names ("\w") but ascii matching for
83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	60 # number literals.
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	61 Whitespace = r'[ \f\t]*'
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	62 Comment = r'#[^\r\n]*'
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	63 Ignore = Whitespace + any(r'\\\r?\n' + Whitespace) + maybe(Comment)
64318 a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	64 Name = r'\w+'
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	65
48769 83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	66 Hexnumber = r'0[xX][0-9a-fA-F]+'
44782 ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from Georg Brandl <georg@python.org> parents: 43833 diff changeset	67 Binnumber = r'0[bB][01]+'
ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from Georg Brandl <georg@python.org> parents: 43833 diff changeset	68 Octnumber = r'0[oO][0-7]+'
48769 83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	69 Decnumber = r'(?:0+\|[1-9][0-9]*)'
42084 229c28cb9afb Merged revisions 55817-55961 via svnmerge from Guido van Rossum <guido@python.org> parents: 41717 diff changeset	70 Intnumber = group(Hexnumber, Binnumber, Octnumber, Decnumber)
48769 83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	71 Exponent = r'[eE][-+]?[0-9]+'
83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	72 Pointfloat = group(r'[0-9]+\.[0-9]*', r'\.[0-9]+') + maybe(Exponent)
83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	73 Expfloat = r'[0-9]+' + Exponent
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	74 Floatnumber = group(Pointfloat, Expfloat)
48769 83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	75 Imagnumber = group(r'[0-9]+[jJ]', Floatnumber + r'[jJ]')
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	76 Number = group(Imagnumber, Floatnumber, Intnumber)
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	77
15080 0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	78 # Tail end of ' string.
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	79 Single = r"[^'\\](?:\\.[^'\\])*'"
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	80 # Tail end of " string.
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	81 Double = r'[^"\\](?:\\.[^"\\])*"'
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	82 # Tail end of ''' string.
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	83 Single3 = r"[^'\\](?:(?:\\.\|'(?!''))[^'\\])*'''"
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	84 # Tail end of """ string.
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	85 Double3 = r'[^"\\](?:(?:\\.\|"(?!""))[^"\\])*"""'
43833 ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	86 Triple = group("[bB]?[rR]?'''", '[bB]?[rR]?"""')
15080 0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	87 # Single-line ' or " string.
43833 ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	88 String = group(r"[bB]?[rR]?'[^\n'\\](?:\\.[^\n'\\])*'",
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	89 r'[bB]?[rR]?"[^\n"\\](?:\\.[^\n"\\])*"')
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	90
15080 0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	91 # Because of leftmost-then-longest match semantics, be sure to put the
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	92 # longest operators first (e.g., if = came before ==, == would get
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	93 # recognized as two instances of =).
39648 efe3f2c0da5e Killed the <> operator. You must now use !=. Guido van Rossum <guido@python.org> parents: 39610 diff changeset	94 Operator = group(r"\\=?", r">>=?", r"<<=?", r"!=",
40529 54cd47788811 PEP 3107 - Function Annotations thanks to Tony Lownds Neal Norwitz <nnorwitz@gmail.com> parents: 40450 diff changeset	95 r"//=?", r"->",
15080 0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	96 r"[+\-*/%&\|^=<>]=?",
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	97 r"~")
14166 e9c2f830f9ba Update for augmented assignment, tested & approved by Guido. Thomas Wouters <thomas@python.org> parents: 14021 diff changeset	98
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	99 Bracket = '[][(){}]'
41230 b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis. Georg Brandl <georg@python.org> parents: 40775 diff changeset	100 Special = group(r'\r?\n', r'\.\.\.', r'[:;.,@]')
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	101 Funny = group(Operator, Bracket, Special)
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	102
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	103 PlainToken = group(Number, Funny, String, Name)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	104 Token = Ignore + PlainToken
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	105
15080 0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	106 # First (or only) line of ' or " string.
43833 ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	107 ContStr = group(r"[bB]?[rR]?'[^\n'\\](?:\\.[^\n'\\])*" +
15898 8366fcfb6090 Add tokenizer support and tests for u'', U"", uR'', Ur"", etc. Ka-Ping Yee <ping@zesty.ca> parents: 15873 diff changeset	108 group("'", r'\\\r?\n'),
43833 ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	109 r'[bB]?[rR]?"[^\n"\\](?:\\.[^\n"\\])*' +
15898 8366fcfb6090 Add tokenizer support and tests for u'', U"", uR'', Ur"", etc. Ka-Ping Yee <ping@zesty.ca> parents: 15873 diff changeset	110 group('"', r'\\\r?\n'))
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	111 PseudoExtras = group(r'\\\r?\n', Comment, Triple)
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	112 PseudoToken = Whitespace + group(PseudoExtras, Number, Funny, ContStr, Name)
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	113
64318 a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	114 def _compile(expr):
a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	115 return re.compile(expr, re.UNICODE)
a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	116
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	117 tokenprog, pseudoprog, single3prog, double3prog = map(
64318 a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	118 _compile, (Token, PseudoToken, Single3, Double3))
a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	119 endprogs = {"'": _compile(Single), '"': _compile(Double),
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	120 "'''": single3prog, '"""': double3prog,
6532 d369629e8c27 New, fixed version with proper r"..." and R"..." support from Ka-Ping. Guido van Rossum <guido@python.org> parents: 6531 diff changeset	121 "r'''": single3prog, 'r"""': double3prog,
43833 ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	122 "b'''": single3prog, 'b"""': double3prog,
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	123 "br'''": single3prog, 'br"""': double3prog,
15898 8366fcfb6090 Add tokenizer support and tests for u'', U"", uR'', Ur"", etc. Ka-Ping Yee <ping@zesty.ca> parents: 15873 diff changeset	124 "R'''": single3prog, 'R"""': double3prog,
43833 ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	125 "B'''": single3prog, 'B"""': double3prog,
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	126 "bR'''": single3prog, 'bR"""': double3prog,
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	127 "Br'''": single3prog, 'Br"""': double3prog,
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	128 "BR'''": single3prog, 'BR"""': double3prog,
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	129 'r': None, 'R': None, 'b': None, 'B': None}
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	130
25016 0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	131 triple_quoted = {}
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	132 for t in ("'''", '"""',
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	133 "r'''", 'r"""', "R'''", 'R"""',
43833 ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	134 "b'''", 'b"""', "B'''", 'B"""',
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	135 "br'''", 'br"""', "Br'''", 'Br"""',
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	136 "bR'''", 'bR"""', "BR'''", 'BR"""'):
25016 0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	137 triple_quoted[t] = t
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	138 single_quoted = {}
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	139 for t in ("'", '"',
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	140 "r'", 'r"', "R'", 'R"',
43833 ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	141 "b'", 'b"', "B'", 'B"',
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	142 "br'", 'br"', "Br'", 'Br"',
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	143 "bR'", 'bR"', "BR'", 'BR"' ):
25016 0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	144 single_quoted[t] = t
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	145
64318 a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	146 del _compile
a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	147
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	148 tabsize = 8
14021 5b34b70ecf22 Convert some old-style string exceptions to class exceptions. Fred Drake <fdrake@acm.org> parents: 7839 diff changeset	149
17178 707f95d05e69 Provide a StopTokenizing exception for conveniently exiting the loop. Ka-Ping Yee <ping@zesty.ca> parents: 16901 diff changeset	150 class TokenError(Exception): pass
707f95d05e69 Provide a StopTokenizing exception for conveniently exiting the loop. Ka-Ping Yee <ping@zesty.ca> parents: 16901 diff changeset	151
707f95d05e69 Provide a StopTokenizing exception for conveniently exiting the loop. Ka-Ping Yee <ping@zesty.ca> parents: 16901 diff changeset	152 class StopTokenizing(Exception): pass
14021 5b34b70ecf22 Convert some old-style string exceptions to class exceptions. Fred Drake <fdrake@acm.org> parents: 7839 diff changeset	153
18148 7bc473b2c654 Merging the gen-branch into the main line, at Guido's direction. Yay! Tim Peters <tim.peters@gmail.com> parents: 17178 diff changeset	154
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	155 class Untokenizer:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	156
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	157 def __init__(self):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	158 self.tokens = []
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	159 self.prev_row = 1
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	160 self.prev_col = 0
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	161 self.encoding = None
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	162
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	163 def add_whitespace(self, start):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	164 row, col = start
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	165 assert row <= self.prev_row
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	166 col_offset = col - self.prev_col
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	167 if col_offset:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	168 self.tokens.append(" " * col_offset)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	169
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	170 def untokenize(self, iterable):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	171 for t in iterable:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	172 if len(t) == 2:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	173 self.compat(t, iterable)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	174 break
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	175 tok_type, token, start, end, line = t
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	176 if tok_type == ENCODING:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	177 self.encoding = token
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	178 continue
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	179 self.add_whitespace(start)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	180 self.tokens.append(token)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	181 self.prev_row, self.prev_col = end
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	182 if tok_type in (NEWLINE, NL):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	183 self.prev_row += 1
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	184 self.prev_col = 0
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	185 return "".join(self.tokens)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	186
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	187 def compat(self, token, iterable):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	188 startline = False
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	189 indents = []
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	190 toks_append = self.tokens.append
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	191 toknum, tokval = token
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	192
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	193 if toknum in (NAME, NUMBER):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	194 tokval += ' '
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	195 if toknum in (NEWLINE, NL):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	196 startline = True
46145 da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	197 prevstring = False
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	198 for tok in iterable:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	199 toknum, tokval = tok[:2]
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	200 if toknum == ENCODING:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	201 self.encoding = tokval
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	202 continue
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	203
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	204 if toknum in (NAME, NUMBER):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	205 tokval += ' '
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	206
46145 da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	207 # Insert a space between two consecutive strings
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	208 if toknum == STRING:
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	209 if prevstring:
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	210 tokval = ' ' + tokval
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	211 prevstring = True
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	212 else:
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	213 prevstring = False
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	214
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	215 if toknum == INDENT:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	216 indents.append(tokval)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	217 continue
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	218 elif toknum == DEDENT:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	219 indents.pop()
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	220 continue
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	221 elif toknum in (NEWLINE, NL):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	222 startline = True
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	223 elif startline and indents:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	224 toks_append(indents[-1])
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	225 startline = False
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	226 toks_append(tokval)
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	227
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	228
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	229 def untokenize(iterable):
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	230 """Transform tokens back into Python source code.
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	231 It returns a bytes object, encoded using the ENCODING
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	232 token, which is the first token sequence output by tokenize.
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	233
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	234 Each element returned by the iterable must be a token sequence
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	235 with at least two elements, a token number and token value. If
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	236 only two tokens are passed, the resulting output is poor.
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	237
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	238 Round-trip invariant for full input:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	239 Untokenized source will match input source exactly
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	240
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	241 Round-trip invariant for limited intput:
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	242 # Output bytes will tokenize the back to the input
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	243 t1 = [tok[:2] for tok in tokenize(f.readline)]
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	244 newcode = untokenize(t1)
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	245 readline = BytesIO(newcode).readline
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	246 t2 = [tok[:2] for tok in tokenize(readline)]
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	247 assert t1 == t2
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	248 """
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	249 ut = Untokenizer()
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	250 out = ut.untokenize(iterable)
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	251 if ut.encoding is not None:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	252 out = out.encode(ut.encoding)
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	253 return out
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	254
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	255
56434 12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	256 def _get_normal_name(orig_enc):
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	257 """Imitates get_normal_name in tokenizer.c."""
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	258 # Only care about the first 12 characters.
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	259 enc = orig_enc[:12].lower().replace("_", "-")
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	260 if enc == "utf-8" or enc.startswith("utf-8-"):
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	261 return "utf-8"
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	262 if enc in ("latin-1", "iso-8859-1", "iso-latin-1") or \
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	263 enc.startswith(("latin-1-", "iso-8859-1-", "iso-latin-1-")):
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	264 return "iso-8859-1"
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	265 return orig_enc
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	266
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	267 def detect_encoding(readline):
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	268 """
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	269 The detect_encoding() function is used to detect the encoding that should
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	270 be used to decode a Python source file. It requires one argment, readline,
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	271 in the same way as the tokenize() generator.
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	272
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	273 It will call readline a maximum of twice, and return the encoding used
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	274 (as a string) and a list of any lines (left as bytes) it has read in.
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	275
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	276 It detects the encoding from the presence of a utf-8 bom or an encoding
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	277 cookie as specified in pep-0263. If both a bom and a cookie are present,
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	278 but disagree, a SyntaxError will be raised. If the encoding cookie is an
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	279 invalid charset, raise a SyntaxError. Note that if a utf-8 bom is found,
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	280 'utf-8-sig' is returned.
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	281
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	282 If no encoding is specified, then the default of 'utf-8' will be returned.
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	283 """
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	284 bom_found = False
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	285 encoding = None
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	286 default = 'utf-8'
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	287 def read_or_stop():
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	288 try:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	289 return readline()
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	290 except StopIteration:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	291 return b''
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	292
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	293 def find_cookie(line):
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	294 try:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	295 line_string = line.decode('ascii')
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	296 except UnicodeDecodeError:
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	297 return None
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	298
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	299 matches = cookie_re.findall(line_string)
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	300 if not matches:
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	301 return None
56434 12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	302 encoding = _get_normal_name(matches[0])
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	303 try:
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	304 codec = lookup(encoding)
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	305 except LookupError:
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	306 # This behaviour mimics the Python interpreter
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	307 raise SyntaxError("unknown encoding: " + encoding)
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	308
59701 15b87ea1413d fix for files with coding cookies and BOMs Benjamin Peterson <benjamin@python.org> parents: 59700 diff changeset	309 if bom_found:
15b87ea1413d fix for files with coding cookies and BOMs Benjamin Peterson <benjamin@python.org> parents: 59700 diff changeset	310 if codec.name != 'utf-8':
15b87ea1413d fix for files with coding cookies and BOMs Benjamin Peterson <benjamin@python.org> parents: 59700 diff changeset	311 # This behaviour mimics the Python interpreter
15b87ea1413d fix for files with coding cookies and BOMs Benjamin Peterson <benjamin@python.org> parents: 59700 diff changeset	312 raise SyntaxError('encoding problem: utf-8')
15b87ea1413d fix for files with coding cookies and BOMs Benjamin Peterson <benjamin@python.org> parents: 59700 diff changeset	313 encoding += '-sig'
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	314 return encoding
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	315
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	316 first = read_or_stop()
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	317 if first.startswith(BOM_UTF8):
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	318 bom_found = True
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	319 first = first[3:]
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	320 default = 'utf-8-sig'
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	321 if not first:
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	322 return default, []
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	323
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	324 encoding = find_cookie(first)
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	325 if encoding:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	326 return encoding, [first]
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	327
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	328 second = read_or_stop()
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	329 if not second:
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	330 return default, [first]
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	331
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	332 encoding = find_cookie(second)
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	333 if encoding:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	334 return encoding, [first, second]
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	335
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	336 return default, [first, second]
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	337
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	338
66097 306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	339 def open(filename):
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	340 """Open a file in read only mode using the encoding detected by
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	341 detect_encoding().
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	342 """
68096 ea260d393cde Issue #11074: Make 'tokenize' so it can be reloaded. Brett Cannon <bcannon@gmail.com> parents: 66148 diff changeset	343 buffer = builtins.open(filename, 'rb')
66097 306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	344 encoding, lines = detect_encoding(buffer.readline)
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	345 buffer.seek(0)
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	346 text = TextIOWrapper(buffer, encoding, line_buffering=True)
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	347 text.mode = 'r'
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	348 return text
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	349
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	350
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	351 def tokenize(readline):
23455 68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	352 """
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	353 The tokenize() generator requires one argment, readline, which
23455 68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	354 must be a callable object which provides the same interface as the
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	355 readline() method of built-in file objects. Each call to the function
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	356 should return one line of input as bytes. Alternately, readline
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	357 can be a callable function terminating with StopIteration:
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	358 readline = open(myfile, 'rb').__next__ # Example of alternate readline
23521 a90faa5f9417 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 23455 diff changeset	359
23455 68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	360 The generator produces 5-tuples with these members: the token type; the
68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	361 token string; a 2-tuple (srow, scol) of ints specifying the row and
68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	362 column where the token begins in the source; a 2-tuple (erow, ecol) of
68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	363 ints specifying the row and column where the token ends in the source;
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	364 and the line on which the token was found. The line passed is the
23521 a90faa5f9417 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 23455 diff changeset	365 logical line; continuation lines are included.
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	366
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	367 The first token sequence will always be an ENCODING token
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	368 which tells you which encoding was used to decode the bytes stream.
23455 68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	369 """
57235 555045541794 simply by using itertools.chain() Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	370 # This import is here to avoid problems when the itertools module is not
555045541794 simply by using itertools.chain() Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	371 # built yet and tokenize is imported.
57237 94f2705def5c use some more itertools magic to make '' be yielded after readline is done Benjamin Peterson <benjamin@python.org> parents: 57235 diff changeset	372 from itertools import chain, repeat
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	373 encoding, consumed = detect_encoding(readline)
57237 94f2705def5c use some more itertools magic to make '' be yielded after readline is done Benjamin Peterson <benjamin@python.org> parents: 57235 diff changeset	374 rl_gen = iter(readline, b"")
94f2705def5c use some more itertools magic to make '' be yielded after readline is done Benjamin Peterson <benjamin@python.org> parents: 57235 diff changeset	375 empty = repeat(b"")
94f2705def5c use some more itertools magic to make '' be yielded after readline is done Benjamin Peterson <benjamin@python.org> parents: 57235 diff changeset	376 return _tokenize(chain(consumed, rl_gen, empty).__next__, encoding)
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	377
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	378
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	379 def _tokenize(readline, encoding):
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	380 lnum = parenlev = continued = 0
64318 a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	381 numchars = '0123456789'
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	382 contstr, needcont = '', 0
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	383 contline = None
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	384 indents = [0]
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	385
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	386 if encoding is not None:
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	387 if encoding == "utf-8-sig":
72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	388 # BOM will already have been stripped.
72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	389 encoding = "utf-8"
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	390 yield TokenInfo(ENCODING, encoding, (0, 0), (0, 0), '')
47481 0d7ccb50007c use the more idomatic (and Py3k faster) while True Benjamin Peterson <benjamin@python.org> parents: 46145 diff changeset	391 while True: # loop over lines in stream
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	392 try:
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	393 line = readline()
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	394 except StopIteration:
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	395 line = b''
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	396
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	397 if encoding is not None:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	398 line = line.decode(encoding)
57210 a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	399 lnum += 1
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	400 pos, max = 0, len(line)
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	401
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	402 if contstr: # continued string
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	403 if not line:
43020 907a45df1295 Raise statement normalization in Lib/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	404 raise TokenError("EOF in multi-line string", strstart)
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	405 endmatch = endprog.match(line)
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	406 if endmatch:
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	407 pos = end = endmatch.end(0)
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	408 yield TokenInfo(STRING, contstr + line[:end],
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	409 strstart, (lnum, end), contline + line)
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	410 contstr, needcont = '', 0
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	411 contline = None
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	412 elif needcont and line[-2:] != '\\\n' and line[-3:] != '\\\r\n':
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	413 yield TokenInfo(ERRORTOKEN, contstr + line,
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	414 strstart, (lnum, len(line)), contline)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	415 contstr = ''
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	416 contline = None
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	417 continue
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	418 else:
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	419 contstr = contstr + line
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	420 contline = contline + line
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	421 continue
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	422
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	423 elif parenlev == 0 and not continued: # new statement
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	424 if not line: break
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	425 column = 0
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	426 while pos < max: # measure leading whitespace
57210 a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	427 if line[pos] == ' ':
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	428 column += 1
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	429 elif line[pos] == '\t':
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	430 column = (column//tabsize + 1)*tabsize
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	431 elif line[pos] == '\f':
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	432 column = 0
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	433 else:
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	434 break
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	435 pos += 1
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	436 if pos == max:
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	437 break
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	438
1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	439 if line[pos] in '#\r\n': # skip comments or blank lines
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	440 if line[pos] == '#':
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	441 comment_token = line[pos:].rstrip('\r\n')
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	442 nl_pos = pos + len(comment_token)
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	443 yield TokenInfo(COMMENT, comment_token,
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	444 (lnum, pos), (lnum, pos + len(comment_token)), line)
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	445 yield TokenInfo(NL, line[nl_pos:],
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	446 (lnum, nl_pos), (lnum, len(line)), line)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	447 else:
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	448 yield TokenInfo((NL, COMMENT)[line[pos] == '#'], line[pos:],
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	449 (lnum, pos), (lnum, len(line)), line)
1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	450 continue
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	451
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	452 if column > indents[-1]: # count indents or dedents
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	453 indents.append(column)
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	454 yield TokenInfo(INDENT, line[:pos], (lnum, 0), (lnum, pos), line)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	455 while column < indents[-1]:
35119 04868959f1fe SF bug #1224621: tokenize module does not detect inconsistent dedents Raymond Hettinger <python@rcn.com> parents: 35058 diff changeset	456 if column not in indents:
04868959f1fe SF bug #1224621: tokenize module does not detect inconsistent dedents Raymond Hettinger <python@rcn.com> parents: 35058 diff changeset	457 raise IndentationError(
39610 3674f20ac781 Merge current trunk into p3yk. This includes the PyNumber_Index API change, Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	458 "unindent does not match any outer indentation level",
3674f20ac781 Merge current trunk into p3yk. This includes the PyNumber_Index API change, Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	459 ("<tokenize>", lnum, pos, line))
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	460 indents = indents[:-1]
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	461 yield TokenInfo(DEDENT, '', (lnum, pos), (lnum, pos), line)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	462
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	463 else: # continued statement
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	464 if not line:
43020 907a45df1295 Raise statement normalization in Lib/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	465 raise TokenError("EOF in multi-line statement", (lnum, 0))
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	466 continued = 0
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	467
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	468 while pos < max:
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	469 pseudomatch = pseudoprog.match(line, pos)
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	470 if pseudomatch: # scan for tokens
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	471 start, end = pseudomatch.span(1)
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	472 spos, epos, pos = (lnum, start), (lnum, end), end
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	473 token, initial = line[start:end], line[start]
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	474
41230 b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis. Georg Brandl <georg@python.org> parents: 40775 diff changeset	475 if (initial in numchars or # ordinary number
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis. Georg Brandl <georg@python.org> parents: 40775 diff changeset	476 (initial == '.' and token != '.' and token != '...')):
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	477 yield TokenInfo(NUMBER, token, spos, epos, line)
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	478 elif initial in '\r\n':
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	479 yield TokenInfo(NL if parenlev > 0 else NEWLINE,
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	480 token, spos, epos, line)
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	481 elif initial == '#':
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	482 assert not token.endswith("\n")
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	483 yield TokenInfo(COMMENT, token, spos, epos, line)
25016 0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	484 elif token in triple_quoted:
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	485 endprog = endprogs[token]
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	486 endmatch = endprog.match(line, pos)
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	487 if endmatch: # all on one line
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	488 pos = endmatch.end(0)
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	489 token = line[start:pos]
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	490 yield TokenInfo(STRING, token, spos, (lnum, pos), line)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	491 else:
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	492 strstart = (lnum, start) # multiple lines
1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	493 contstr = line[start:]
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	494 contline = line
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	495 break
25016 0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	496 elif initial in single_quoted or \
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	497 token[:2] in single_quoted or \
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	498 token[:3] in single_quoted:
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	499 if token[-1] == '\n': # continued string
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	500 strstart = (lnum, start)
15898 8366fcfb6090 Add tokenizer support and tests for u'', U"", uR'', Ur"", etc. Ka-Ping Yee <ping@zesty.ca> parents: 15873 diff changeset	501 endprog = (endprogs[initial] or endprogs[token[1]] or
8366fcfb6090 Add tokenizer support and tests for u'', U"", uR'', Ur"", etc. Ka-Ping Yee <ping@zesty.ca> parents: 15873 diff changeset	502 endprogs[token[2]])
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	503 contstr, needcont = line[start:], 1
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	504 contline = line
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	505 break
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	506 else: # ordinary string
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	507 yield TokenInfo(STRING, token, spos, epos, line)
64318 a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	508 elif initial.isidentifier(): # ordinary name
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	509 yield TokenInfo(NAME, token, spos, epos, line)
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	510 elif initial == '\\': # continued stmt
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	511 continued = 1
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	512 else:
57210 a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	513 if initial in '([{':
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	514 parenlev += 1
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	515 elif initial in ')]}':
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	516 parenlev -= 1
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	517 yield TokenInfo(OP, token, spos, epos, line)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	518 else:
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	519 yield TokenInfo(ERRORTOKEN, line[pos],
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	520 (lnum, pos), (lnum, pos+1), line)
57210 a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	521 pos += 1
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	522
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	523 for indent in indents[1:]: # pop remaining indent levels
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	524 yield TokenInfo(DEDENT, '', (lnum, 0), (lnum, 0), '')
7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	525 yield TokenInfo(ENDMARKER, '', (lnum, 0), (lnum, 0), '')
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	526
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	527
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	528 # An undocumented, backwards compatible, API for all the places in the standard
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	529 # library that expect to be able to use tokenize with strings
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	530 def generate_tokens(readline):
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	531 return _tokenize(readline, None)
64577 2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	532
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	533 if __name__ == "__main__":
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	534 # Quick sanity check
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	535 s = b'''def parseline(self, line):
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	536 """Parse the line into a command name and a string containing
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	537 the arguments. Returns a tuple containing (command, args, line).
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	538 'command' and 'args' may be None if the line couldn't be parsed.
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	539 """
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	540 line = line.strip()
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	541 if not line:
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	542 return None, None, line
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	543 elif line[0] == '?':
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	544 line = 'help ' + line[1:]
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	545 elif line[0] == '!':
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	546 if hasattr(self, 'do_shell'):
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	547 line = 'shell ' + line[1:]
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	548 else:
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	549 return None, None, line
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	550 i, n = 0, len(line)
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	551 while i < n and line[i] in self.identchars: i = i+1
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	552 cmd, arg = line[:i], line[i:].strip()
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	553 return cmd, arg, line
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	554 '''
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	555 for tok in tokenize(iter(s.splitlines()).__next__):
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	556 print(tok)

Mercurial > cpython-lru_cache-weakref

annotate Lib/tokenize.py @ 74135:9c19df6c8ea0 3.2