cpython-lru_cache-weakref: Lib/tokenize.py annotate

annotate Lib/tokenize.py @ 73798:6cde416ef03b

Credit Per Øyvind Karlsen for the initial implementation of the lzma module (issue #6715).

author	Nadeem Vawda <nadeem.vawda@gmail.com>
date	Thu, 01 Dec 2011 01:18:27 +0200
parents	df950158dc33
children	f4976fa6e830

rev	line source
5173 ff4f8fa8a273 Fixed doc string, added __version__, fixed 1 bug. Guido van Rossum <guido@python.org> parents: 5172 diff changeset	1 """Tokenization help for Python programs.
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	2
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	3 tokenize(readline) is a generator that breaks a stream of bytes into
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	4 Python tokens. It decodes the bytes according to PEP-0263 for
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	5 determining source file encoding.
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	6
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	7 It accepts a readline-like method which is called repeatedly to get the
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	8 next line of input (or b"" for EOF). It generates 5-tuples with these
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	9 members:
18351 44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	10
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	11 the token type (see token.py)
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	12 the token (a string)
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	13 the starting (row, column) indices of the token (a 2-tuple of ints)
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	14 the ending (row, column) indices of the token (a 2-tuple of ints)
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	15 the original line (string)
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	16
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	17 It is designed to match the working of the Python tokenizer exactly, except
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	18 that it produces COMMENT tokens for comments and gives type OP for all
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	19 operators. Additionally, all token lists start with an ENCODING token
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	20 which tells you which encoding was used to decode the bytes stream.
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	21 """
5173 ff4f8fa8a273 Fixed doc string, added __version__, fixed 1 bug. Guido van Rossum <guido@python.org> parents: 5172 diff changeset	22
16899 320fa50a1a7d Add __author__ and __credits__ variables. Ka-Ping Yee <ping@zesty.ca> parents: 16889 diff changeset	23 __author__ = 'Ka-Ping Yee <ping@lfw.org>'
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	24 __credits__ = ('GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, '
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	25 'Skip Montanaro, Raymond Hettinger, Trent Nelson, '
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	26 'Michael Foord')
68096 ea260d393cde Issue #11074: Make 'tokenize' so it can be reloaded. Brett Cannon <bcannon@gmail.com> parents: 66148 diff changeset	27 import builtins
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	28 import re
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	29 import sys
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	30 from token import *
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	31 from codecs import lookup, BOM_UTF8
64581 fea86c5e3385 Experiment: Let collections.namedtuple() do the work. This should work now that _collections is pre-built. The buildbots will tell us shortly. Raymond Hettinger <python@rcn.com> parents: 64577 diff changeset	32 import collections
66097 306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	33 from io import TextIOWrapper
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	34 cookie_re = re.compile("coding[:=]\s*([-\w.]+)")
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	35
16889 0feb63490093 final round of __all__ lists (I hope) - skipped urllib2 because Moshe may be Skip Montanaro <skip@pobox.com> parents: 16514 diff changeset	36 import token
66148 98bd3ca1ed86 Issue #10386: Added __all__ to token module; this simplifies importing Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66097 diff changeset	37 __all__ = token.__all__ + ["COMMENT", "tokenize", "detect_encoding",
98bd3ca1ed86 Issue #10386: Added __all__ to token module; this simplifies importing Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66097 diff changeset	38 "NL", "untokenize", "ENCODING", "TokenInfo"]
16889 0feb63490093 final round of __all__ lists (I hope) - skipped urllib2 because Moshe may be Skip Montanaro <skip@pobox.com> parents: 16514 diff changeset	39 del token
0feb63490093 final round of __all__ lists (I hope) - skipped urllib2 because Moshe may be Skip Montanaro <skip@pobox.com> parents: 16514 diff changeset	40
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	41 COMMENT = N_TOKENS
1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	42 tok_name[COMMENT] = 'COMMENT'
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	43 NL = N_TOKENS + 1
d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	44 tok_name[NL] = 'NL'
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	45 ENCODING = N_TOKENS + 2
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	46 tok_name[ENCODING] = 'ENCODING'
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	47 N_TOKENS += 3
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	48
64581 fea86c5e3385 Experiment: Let collections.namedtuple() do the work. This should work now that _collections is pre-built. The buildbots will tell us shortly. Raymond Hettinger <python@rcn.com> parents: 64577 diff changeset	49 class TokenInfo(collections.namedtuple('TokenInfo', 'type string start end line')):
53597 5f81c45bdf6d Remove dependency on the collections module. Raymond Hettinger <python@rcn.com> parents: 53584 diff changeset	50 def __repr__(self):
64582 71b264e1bb71 A little bit more readable repr method. Raymond Hettinger <python@rcn.com> parents: 64581 diff changeset	51 annotated_type = '%d (%s)' % (self.type, tok_name[self.type])
71b264e1bb71 A little bit more readable repr method. Raymond Hettinger <python@rcn.com> parents: 64581 diff changeset	52 return ('TokenInfo(type=%s, string=%r, start=%r, end=%r, line=%r)' %
71b264e1bb71 A little bit more readable repr method. Raymond Hettinger <python@rcn.com> parents: 64581 diff changeset	53 self._replace(type=annotated_type))
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	54
16514 90c600a819e9 String method conversion. Eric S. Raymond <esr@thyrsus.com> parents: 15898 diff changeset	55 def group(*choices): return '(' + '\|'.join(choices) + ')'
28089 14f65d661084 Get rid of many apply() calls. Guido van Rossum <guido@python.org> parents: 26032 diff changeset	56 def any(choices): return group(choices) + '*'
14f65d661084 Get rid of many apply() calls. Guido van Rossum <guido@python.org> parents: 26032 diff changeset	57 def maybe(choices): return group(choices) + '?'
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	58
48769 83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	59 # Note: we use unicode matching for names ("\w") but ascii matching for
83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	60 # number literals.
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	61 Whitespace = r'[ \f\t]*'
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	62 Comment = r'#[^\r\n]*'
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	63 Ignore = Whitespace + any(r'\\\r?\n' + Whitespace) + maybe(Comment)
64318 a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	64 Name = r'\w+'
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	65
48769 83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	66 Hexnumber = r'0[xX][0-9a-fA-F]+'
44782 ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from Georg Brandl <georg@python.org> parents: 43833 diff changeset	67 Binnumber = r'0[bB][01]+'
ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from Georg Brandl <georg@python.org> parents: 43833 diff changeset	68 Octnumber = r'0[oO][0-7]+'
48769 83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	69 Decnumber = r'(?:0+\|[1-9][0-9]*)'
42084 229c28cb9afb Merged revisions 55817-55961 via svnmerge from Guido van Rossum <guido@python.org> parents: 41717 diff changeset	70 Intnumber = group(Hexnumber, Binnumber, Octnumber, Decnumber)
48769 83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	71 Exponent = r'[eE][-+]?[0-9]+'
83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	72 Pointfloat = group(r'[0-9]+\.[0-9]*', r'\.[0-9]+') + maybe(Exponent)
83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	73 Expfloat = r'[0-9]+' + Exponent
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	74 Floatnumber = group(Pointfloat, Expfloat)
48769 83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	75 Imagnumber = group(r'[0-9]+[jJ]', Floatnumber + r'[jJ]')
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	76 Number = group(Imagnumber, Floatnumber, Intnumber)
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	77
15080 0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	78 # Tail end of ' string.
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	79 Single = r"[^'\\](?:\\.[^'\\])*'"
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	80 # Tail end of " string.
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	81 Double = r'[^"\\](?:\\.[^"\\])*"'
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	82 # Tail end of ''' string.
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	83 Single3 = r"[^'\\](?:(?:\\.\|'(?!''))[^'\\])*'''"
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	84 # Tail end of """ string.
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	85 Double3 = r'[^"\\](?:(?:\\.\|"(?!""))[^"\\])*"""'
43833 ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	86 Triple = group("[bB]?[rR]?'''", '[bB]?[rR]?"""')
15080 0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	87 # Single-line ' or " string.
43833 ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	88 String = group(r"[bB]?[rR]?'[^\n'\\](?:\\.[^\n'\\])*'",
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	89 r'[bB]?[rR]?"[^\n"\\](?:\\.[^\n"\\])*"')
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	90
15080 0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	91 # Because of leftmost-then-longest match semantics, be sure to put the
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	92 # longest operators first (e.g., if = came before ==, == would get
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	93 # recognized as two instances of =).
39648 efe3f2c0da5e Killed the <> operator. You must now use !=. Guido van Rossum <guido@python.org> parents: 39610 diff changeset	94 Operator = group(r"\\=?", r">>=?", r"<<=?", r"!=",
40529 54cd47788811 PEP 3107 - Function Annotations thanks to Tony Lownds Neal Norwitz <nnorwitz@gmail.com> parents: 40450 diff changeset	95 r"//=?", r"->",
15080 0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	96 r"[+\-*/%&\|^=<>]=?",
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	97 r"~")
14166 e9c2f830f9ba Update for augmented assignment, tested & approved by Guido. Thomas Wouters <thomas@python.org> parents: 14021 diff changeset	98
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	99 Bracket = '[][(){}]'
41230 b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis. Georg Brandl <georg@python.org> parents: 40775 diff changeset	100 Special = group(r'\r?\n', r'\.\.\.', r'[:;.,@]')
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	101 Funny = group(Operator, Bracket, Special)
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	102
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	103 PlainToken = group(Number, Funny, String, Name)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	104 Token = Ignore + PlainToken
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	105
15080 0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	106 # First (or only) line of ' or " string.
43833 ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	107 ContStr = group(r"[bB]?[rR]?'[^\n'\\](?:\\.[^\n'\\])*" +
15898 8366fcfb6090 Add tokenizer support and tests for u'', U"", uR'', Ur"", etc. Ka-Ping Yee <ping@zesty.ca> parents: 15873 diff changeset	108 group("'", r'\\\r?\n'),
43833 ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	109 r'[bB]?[rR]?"[^\n"\\](?:\\.[^\n"\\])*' +
15898 8366fcfb6090 Add tokenizer support and tests for u'', U"", uR'', Ur"", etc. Ka-Ping Yee <ping@zesty.ca> parents: 15873 diff changeset	110 group('"', r'\\\r?\n'))
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	111 PseudoExtras = group(r'\\\r?\n', Comment, Triple)
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	112 PseudoToken = Whitespace + group(PseudoExtras, Number, Funny, ContStr, Name)
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	113
64318 a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	114 def _compile(expr):
a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	115 return re.compile(expr, re.UNICODE)
a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	116
72853 df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	117 endpats = {"'": Single, '"': Double,
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	118 "'''": Single3, '"""': Double3,
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	119 "r'''": Single3, 'r"""': Double3,
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	120 "b'''": Single3, 'b"""': Double3,
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	121 "br'''": Single3, 'br"""': Double3,
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	122 "R'''": Single3, 'R"""': Double3,
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	123 "B'''": Single3, 'B"""': Double3,
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	124 "bR'''": Single3, 'bR"""': Double3,
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	125 "Br'''": Single3, 'Br"""': Double3,
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	126 "BR'''": Single3, 'BR"""': Double3,
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	127 'r': None, 'R': None, 'b': None, 'B': None}
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	128
25016 0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	129 triple_quoted = {}
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	130 for t in ("'''", '"""',
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	131 "r'''", 'r"""', "R'''", 'R"""',
43833 ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	132 "b'''", 'b"""', "B'''", 'B"""',
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	133 "br'''", 'br"""', "Br'''", 'Br"""',
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	134 "bR'''", 'bR"""', "BR'''", 'BR"""'):
25016 0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	135 triple_quoted[t] = t
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	136 single_quoted = {}
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	137 for t in ("'", '"',
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	138 "r'", 'r"', "R'", 'R"',
43833 ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	139 "b'", 'b"', "B'", 'B"',
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	140 "br'", 'br"', "Br'", 'Br"',
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	141 "bR'", 'bR"', "BR'", 'BR"' ):
25016 0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	142 single_quoted[t] = t
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	143
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	144 tabsize = 8
14021 5b34b70ecf22 Convert some old-style string exceptions to class exceptions. Fred Drake <fdrake@acm.org> parents: 7839 diff changeset	145
17178 707f95d05e69 Provide a StopTokenizing exception for conveniently exiting the loop. Ka-Ping Yee <ping@zesty.ca> parents: 16901 diff changeset	146 class TokenError(Exception): pass
707f95d05e69 Provide a StopTokenizing exception for conveniently exiting the loop. Ka-Ping Yee <ping@zesty.ca> parents: 16901 diff changeset	147
707f95d05e69 Provide a StopTokenizing exception for conveniently exiting the loop. Ka-Ping Yee <ping@zesty.ca> parents: 16901 diff changeset	148 class StopTokenizing(Exception): pass
14021 5b34b70ecf22 Convert some old-style string exceptions to class exceptions. Fred Drake <fdrake@acm.org> parents: 7839 diff changeset	149
18148 7bc473b2c654 Merging the gen-branch into the main line, at Guido's direction. Yay! Tim Peters <tim.peters@gmail.com> parents: 17178 diff changeset	150
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	151 class Untokenizer:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	152
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	153 def __init__(self):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	154 self.tokens = []
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	155 self.prev_row = 1
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	156 self.prev_col = 0
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	157 self.encoding = None
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	158
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	159 def add_whitespace(self, start):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	160 row, col = start
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	161 assert row <= self.prev_row
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	162 col_offset = col - self.prev_col
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	163 if col_offset:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	164 self.tokens.append(" " * col_offset)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	165
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	166 def untokenize(self, iterable):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	167 for t in iterable:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	168 if len(t) == 2:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	169 self.compat(t, iterable)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	170 break
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	171 tok_type, token, start, end, line = t
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	172 if tok_type == ENCODING:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	173 self.encoding = token
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	174 continue
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	175 self.add_whitespace(start)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	176 self.tokens.append(token)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	177 self.prev_row, self.prev_col = end
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	178 if tok_type in (NEWLINE, NL):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	179 self.prev_row += 1
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	180 self.prev_col = 0
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	181 return "".join(self.tokens)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	182
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	183 def compat(self, token, iterable):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	184 startline = False
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	185 indents = []
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	186 toks_append = self.tokens.append
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	187 toknum, tokval = token
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	188
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	189 if toknum in (NAME, NUMBER):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	190 tokval += ' '
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	191 if toknum in (NEWLINE, NL):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	192 startline = True
46145 da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	193 prevstring = False
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	194 for tok in iterable:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	195 toknum, tokval = tok[:2]
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	196 if toknum == ENCODING:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	197 self.encoding = tokval
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	198 continue
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	199
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	200 if toknum in (NAME, NUMBER):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	201 tokval += ' '
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	202
46145 da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	203 # Insert a space between two consecutive strings
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	204 if toknum == STRING:
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	205 if prevstring:
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	206 tokval = ' ' + tokval
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	207 prevstring = True
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	208 else:
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	209 prevstring = False
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	210
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	211 if toknum == INDENT:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	212 indents.append(tokval)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	213 continue
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	214 elif toknum == DEDENT:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	215 indents.pop()
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	216 continue
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	217 elif toknum in (NEWLINE, NL):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	218 startline = True
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	219 elif startline and indents:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	220 toks_append(indents[-1])
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	221 startline = False
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	222 toks_append(tokval)
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	223
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	224
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	225 def untokenize(iterable):
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	226 """Transform tokens back into Python source code.
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	227 It returns a bytes object, encoded using the ENCODING
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	228 token, which is the first token sequence output by tokenize.
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	229
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	230 Each element returned by the iterable must be a token sequence
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	231 with at least two elements, a token number and token value. If
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	232 only two tokens are passed, the resulting output is poor.
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	233
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	234 Round-trip invariant for full input:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	235 Untokenized source will match input source exactly
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	236
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	237 Round-trip invariant for limited intput:
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	238 # Output bytes will tokenize the back to the input
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	239 t1 = [tok[:2] for tok in tokenize(f.readline)]
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	240 newcode = untokenize(t1)
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	241 readline = BytesIO(newcode).readline
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	242 t2 = [tok[:2] for tok in tokenize(readline)]
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	243 assert t1 == t2
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	244 """
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	245 ut = Untokenizer()
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	246 out = ut.untokenize(iterable)
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	247 if ut.encoding is not None:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	248 out = out.encode(ut.encoding)
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	249 return out
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	250
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	251
56434 12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	252 def _get_normal_name(orig_enc):
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	253 """Imitates get_normal_name in tokenizer.c."""
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	254 # Only care about the first 12 characters.
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	255 enc = orig_enc[:12].lower().replace("_", "-")
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	256 if enc == "utf-8" or enc.startswith("utf-8-"):
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	257 return "utf-8"
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	258 if enc in ("latin-1", "iso-8859-1", "iso-latin-1") or \
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	259 enc.startswith(("latin-1-", "iso-8859-1-", "iso-latin-1-")):
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	260 return "iso-8859-1"
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	261 return orig_enc
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	262
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	263 def detect_encoding(readline):
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	264 """
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	265 The detect_encoding() function is used to detect the encoding that should
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	266 be used to decode a Python source file. It requires one argment, readline,
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	267 in the same way as the tokenize() generator.
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	268
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	269 It will call readline a maximum of twice, and return the encoding used
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	270 (as a string) and a list of any lines (left as bytes) it has read in.
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	271
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	272 It detects the encoding from the presence of a utf-8 bom or an encoding
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	273 cookie as specified in pep-0263. If both a bom and a cookie are present,
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	274 but disagree, a SyntaxError will be raised. If the encoding cookie is an
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	275 invalid charset, raise a SyntaxError. Note that if a utf-8 bom is found,
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	276 'utf-8-sig' is returned.
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	277
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	278 If no encoding is specified, then the default of 'utf-8' will be returned.
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	279 """
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	280 bom_found = False
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	281 encoding = None
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	282 default = 'utf-8'
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	283 def read_or_stop():
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	284 try:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	285 return readline()
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	286 except StopIteration:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	287 return b''
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	288
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	289 def find_cookie(line):
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	290 try:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	291 line_string = line.decode('ascii')
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	292 except UnicodeDecodeError:
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	293 return None
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	294
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	295 matches = cookie_re.findall(line_string)
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	296 if not matches:
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	297 return None
56434 12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	298 encoding = _get_normal_name(matches[0])
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	299 try:
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	300 codec = lookup(encoding)
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	301 except LookupError:
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	302 # This behaviour mimics the Python interpreter
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	303 raise SyntaxError("unknown encoding: " + encoding)
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	304
59701 15b87ea1413d fix for files with coding cookies and BOMs Benjamin Peterson <benjamin@python.org> parents: 59700 diff changeset	305 if bom_found:
15b87ea1413d fix for files with coding cookies and BOMs Benjamin Peterson <benjamin@python.org> parents: 59700 diff changeset	306 if codec.name != 'utf-8':
15b87ea1413d fix for files with coding cookies and BOMs Benjamin Peterson <benjamin@python.org> parents: 59700 diff changeset	307 # This behaviour mimics the Python interpreter
15b87ea1413d fix for files with coding cookies and BOMs Benjamin Peterson <benjamin@python.org> parents: 59700 diff changeset	308 raise SyntaxError('encoding problem: utf-8')
15b87ea1413d fix for files with coding cookies and BOMs Benjamin Peterson <benjamin@python.org> parents: 59700 diff changeset	309 encoding += '-sig'
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	310 return encoding
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	311
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	312 first = read_or_stop()
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	313 if first.startswith(BOM_UTF8):
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	314 bom_found = True
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	315 first = first[3:]
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	316 default = 'utf-8-sig'
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	317 if not first:
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	318 return default, []
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	319
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	320 encoding = find_cookie(first)
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	321 if encoding:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	322 return encoding, [first]
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	323
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	324 second = read_or_stop()
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	325 if not second:
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	326 return default, [first]
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	327
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	328 encoding = find_cookie(second)
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	329 if encoding:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	330 return encoding, [first, second]
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	331
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	332 return default, [first, second]
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	333
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	334
66097 306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	335 def open(filename):
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	336 """Open a file in read only mode using the encoding detected by
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	337 detect_encoding().
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	338 """
68096 ea260d393cde Issue #11074: Make 'tokenize' so it can be reloaded. Brett Cannon <bcannon@gmail.com> parents: 66148 diff changeset	339 buffer = builtins.open(filename, 'rb')
66097 306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	340 encoding, lines = detect_encoding(buffer.readline)
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	341 buffer.seek(0)
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	342 text = TextIOWrapper(buffer, encoding, line_buffering=True)
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	343 text.mode = 'r'
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	344 return text
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	345
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	346
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	347 def tokenize(readline):
23455 68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	348 """
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	349 The tokenize() generator requires one argment, readline, which
23455 68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	350 must be a callable object which provides the same interface as the
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	351 readline() method of built-in file objects. Each call to the function
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	352 should return one line of input as bytes. Alternately, readline
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	353 can be a callable function terminating with StopIteration:
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	354 readline = open(myfile, 'rb').__next__ # Example of alternate readline
23521 a90faa5f9417 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 23455 diff changeset	355
23455 68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	356 The generator produces 5-tuples with these members: the token type; the
68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	357 token string; a 2-tuple (srow, scol) of ints specifying the row and
68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	358 column where the token begins in the source; a 2-tuple (erow, ecol) of
68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	359 ints specifying the row and column where the token ends in the source;
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	360 and the line on which the token was found. The line passed is the
23521 a90faa5f9417 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 23455 diff changeset	361 logical line; continuation lines are included.
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	362
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	363 The first token sequence will always be an ENCODING token
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	364 which tells you which encoding was used to decode the bytes stream.
23455 68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	365 """
57235 555045541794 simply by using itertools.chain() Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	366 # This import is here to avoid problems when the itertools module is not
555045541794 simply by using itertools.chain() Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	367 # built yet and tokenize is imported.
57237 94f2705def5c use some more itertools magic to make '' be yielded after readline is done Benjamin Peterson <benjamin@python.org> parents: 57235 diff changeset	368 from itertools import chain, repeat
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	369 encoding, consumed = detect_encoding(readline)
57237 94f2705def5c use some more itertools magic to make '' be yielded after readline is done Benjamin Peterson <benjamin@python.org> parents: 57235 diff changeset	370 rl_gen = iter(readline, b"")
94f2705def5c use some more itertools magic to make '' be yielded after readline is done Benjamin Peterson <benjamin@python.org> parents: 57235 diff changeset	371 empty = repeat(b"")
94f2705def5c use some more itertools magic to make '' be yielded after readline is done Benjamin Peterson <benjamin@python.org> parents: 57235 diff changeset	372 return _tokenize(chain(consumed, rl_gen, empty).__next__, encoding)
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	373
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	374
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	375 def _tokenize(readline, encoding):
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	376 lnum = parenlev = continued = 0
64318 a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	377 numchars = '0123456789'
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	378 contstr, needcont = '', 0
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	379 contline = None
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	380 indents = [0]
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	381
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	382 if encoding is not None:
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	383 if encoding == "utf-8-sig":
72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	384 # BOM will already have been stripped.
72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	385 encoding = "utf-8"
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	386 yield TokenInfo(ENCODING, encoding, (0, 0), (0, 0), '')
47481 0d7ccb50007c use the more idomatic (and Py3k faster) while True Benjamin Peterson <benjamin@python.org> parents: 46145 diff changeset	387 while True: # loop over lines in stream
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	388 try:
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	389 line = readline()
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	390 except StopIteration:
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	391 line = b''
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	392
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	393 if encoding is not None:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	394 line = line.decode(encoding)
57210 a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	395 lnum += 1
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	396 pos, max = 0, len(line)
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	397
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	398 if contstr: # continued string
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	399 if not line:
43020 907a45df1295 Raise statement normalization in Lib/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	400 raise TokenError("EOF in multi-line string", strstart)
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	401 endmatch = endprog.match(line)
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	402 if endmatch:
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	403 pos = end = endmatch.end(0)
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	404 yield TokenInfo(STRING, contstr + line[:end],
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	405 strstart, (lnum, end), contline + line)
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	406 contstr, needcont = '', 0
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	407 contline = None
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	408 elif needcont and line[-2:] != '\\\n' and line[-3:] != '\\\r\n':
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	409 yield TokenInfo(ERRORTOKEN, contstr + line,
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	410 strstart, (lnum, len(line)), contline)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	411 contstr = ''
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	412 contline = None
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	413 continue
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	414 else:
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	415 contstr = contstr + line
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	416 contline = contline + line
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	417 continue
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	418
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	419 elif parenlev == 0 and not continued: # new statement
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	420 if not line: break
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	421 column = 0
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	422 while pos < max: # measure leading whitespace
57210 a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	423 if line[pos] == ' ':
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	424 column += 1
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	425 elif line[pos] == '\t':
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	426 column = (column//tabsize + 1)*tabsize
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	427 elif line[pos] == '\f':
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	428 column = 0
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	429 else:
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	430 break
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	431 pos += 1
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	432 if pos == max:
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	433 break
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	434
1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	435 if line[pos] in '#\r\n': # skip comments or blank lines
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	436 if line[pos] == '#':
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	437 comment_token = line[pos:].rstrip('\r\n')
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	438 nl_pos = pos + len(comment_token)
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	439 yield TokenInfo(COMMENT, comment_token,
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	440 (lnum, pos), (lnum, pos + len(comment_token)), line)
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	441 yield TokenInfo(NL, line[nl_pos:],
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	442 (lnum, nl_pos), (lnum, len(line)), line)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	443 else:
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	444 yield TokenInfo((NL, COMMENT)[line[pos] == '#'], line[pos:],
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	445 (lnum, pos), (lnum, len(line)), line)
1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	446 continue
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	447
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	448 if column > indents[-1]: # count indents or dedents
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	449 indents.append(column)
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	450 yield TokenInfo(INDENT, line[:pos], (lnum, 0), (lnum, pos), line)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	451 while column < indents[-1]:
35119 04868959f1fe SF bug #1224621: tokenize module does not detect inconsistent dedents Raymond Hettinger <python@rcn.com> parents: 35058 diff changeset	452 if column not in indents:
04868959f1fe SF bug #1224621: tokenize module does not detect inconsistent dedents Raymond Hettinger <python@rcn.com> parents: 35058 diff changeset	453 raise IndentationError(
39610 3674f20ac781 Merge current trunk into p3yk. This includes the PyNumber_Index API change, Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	454 "unindent does not match any outer indentation level",
3674f20ac781 Merge current trunk into p3yk. This includes the PyNumber_Index API change, Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	455 ("<tokenize>", lnum, pos, line))
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	456 indents = indents[:-1]
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	457 yield TokenInfo(DEDENT, '', (lnum, pos), (lnum, pos), line)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	458
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	459 else: # continued statement
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	460 if not line:
43020 907a45df1295 Raise statement normalization in Lib/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	461 raise TokenError("EOF in multi-line statement", (lnum, 0))
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	462 continued = 0
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	463
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	464 while pos < max:
72853 df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	465 pseudomatch = _compile(PseudoToken).match(line, pos)
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	466 if pseudomatch: # scan for tokens
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	467 start, end = pseudomatch.span(1)
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	468 spos, epos, pos = (lnum, start), (lnum, end), end
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	469 token, initial = line[start:end], line[start]
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	470
41230 b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis. Georg Brandl <georg@python.org> parents: 40775 diff changeset	471 if (initial in numchars or # ordinary number
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis. Georg Brandl <georg@python.org> parents: 40775 diff changeset	472 (initial == '.' and token != '.' and token != '...')):
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	473 yield TokenInfo(NUMBER, token, spos, epos, line)
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	474 elif initial in '\r\n':
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	475 yield TokenInfo(NL if parenlev > 0 else NEWLINE,
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	476 token, spos, epos, line)
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	477 elif initial == '#':
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	478 assert not token.endswith("\n")
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	479 yield TokenInfo(COMMENT, token, spos, epos, line)
25016 0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	480 elif token in triple_quoted:
72853 df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	481 endprog = _compile(endpats[token])
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	482 endmatch = endprog.match(line, pos)
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	483 if endmatch: # all on one line
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	484 pos = endmatch.end(0)
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	485 token = line[start:pos]
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	486 yield TokenInfo(STRING, token, spos, (lnum, pos), line)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	487 else:
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	488 strstart = (lnum, start) # multiple lines
1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	489 contstr = line[start:]
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	490 contline = line
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	491 break
25016 0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	492 elif initial in single_quoted or \
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	493 token[:2] in single_quoted or \
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	494 token[:3] in single_quoted:
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	495 if token[-1] == '\n': # continued string
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	496 strstart = (lnum, start)
72853 df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	497 endprog = _compile(endpats[initial] or
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	498 endpats[token[1]] or
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	499 endpats[token[2]])
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	500 contstr, needcont = line[start:], 1
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	501 contline = line
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	502 break
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	503 else: # ordinary string
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	504 yield TokenInfo(STRING, token, spos, epos, line)
64318 a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	505 elif initial.isidentifier(): # ordinary name
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	506 yield TokenInfo(NAME, token, spos, epos, line)
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	507 elif initial == '\\': # continued stmt
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	508 continued = 1
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	509 else:
57210 a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	510 if initial in '([{':
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	511 parenlev += 1
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	512 elif initial in ')]}':
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	513 parenlev -= 1
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	514 yield TokenInfo(OP, token, spos, epos, line)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	515 else:
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	516 yield TokenInfo(ERRORTOKEN, line[pos],
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	517 (lnum, pos), (lnum, pos+1), line)
57210 a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	518 pos += 1
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	519
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	520 for indent in indents[1:]: # pop remaining indent levels
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	521 yield TokenInfo(DEDENT, '', (lnum, 0), (lnum, 0), '')
7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	522 yield TokenInfo(ENDMARKER, '', (lnum, 0), (lnum, 0), '')
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	523
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	524
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	525 # An undocumented, backwards compatible, API for all the places in the standard
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	526 # library that expect to be able to use tokenize with strings
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	527 def generate_tokens(readline):
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	528 return _tokenize(readline, None)
64577 2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	529
72789 85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	530 def main():
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	531 import argparse
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	532
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	533 # Helper error handling routines
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	534 def perror(message):
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	535 print(message, file=sys.stderr)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	536
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	537 def error(message, filename=None, location=None):
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	538 if location:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	539 args = (filename,) + location + (message,)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	540 perror("%s:%d:%d: error: %s" % args)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	541 elif filename:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	542 perror("%s: error: %s" % (filename, message))
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	543 else:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	544 perror("error: %s" % message)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	545 sys.exit(1)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	546
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	547 # Parse the arguments and options
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	548 parser = argparse.ArgumentParser(prog='python -m tokenize')
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	549 parser.add_argument(dest='filename', nargs='?',
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	550 metavar='filename.py',
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	551 help='the file to tokenize; defaults to stdin')
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	552 args = parser.parse_args()
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	553
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	554 try:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	555 # Tokenize the input
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	556 if args.filename:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	557 filename = args.filename
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	558 with builtins.open(filename, 'rb') as f:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	559 tokens = list(tokenize(f.readline))
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	560 else:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	561 filename = "<stdin>"
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	562 tokens = _tokenize(sys.stdin.readline, None)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	563
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	564 # Output the tokenization
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	565 for token in tokens:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	566 token_range = "%d,%d-%d,%d:" % (token.start + token.end)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	567 print("%-20s%-15s%-15r" %
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	568 (token_range, tok_name[token.type], token.string))
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	569 except IndentationError as err:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	570 line, column = err.args[1][1:3]
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	571 error(err.args[0], filename, (line, column))
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	572 except TokenError as err:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	573 line, column = err.args[1]
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	574 error(err.args[0], filename, (line, column))
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	575 except SyntaxError as err:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	576 error(err, filename)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	577 except IOError as err:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	578 error(err)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	579 except KeyboardInterrupt:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	580 print("interrupted\n")
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	581 except Exception as err:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	582 perror("unexpected error: %s" % err)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	583 raise
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	584
64577 2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	585 if __name__ == "__main__":
72789 85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	586 main()

Mercurial > cpython-lru_cache-weakref

annotate Lib/tokenize.py @ 73798:6cde416ef03b