cpython-lru_cache-weakref: Lib/tokenize.py annotate

annotate Lib/tokenize.py @ 66807:d28126ebd60d

Entries for datetime, callable, and collections.Counter.

author	Raymond Hettinger <python@rcn.com>
date	Thu, 09 Dec 2010 16:41:54 +0000
parents	98bd3ca1ed86
children	ea260d393cde

rev	line source
5173 ff4f8fa8a273 Fixed doc string, added __version__, fixed 1 bug. Guido van Rossum <guido@python.org> parents: 5172 diff changeset	1 """Tokenization help for Python programs.
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	2
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	3 tokenize(readline) is a generator that breaks a stream of bytes into
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	4 Python tokens. It decodes the bytes according to PEP-0263 for
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	5 determining source file encoding.
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	6
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	7 It accepts a readline-like method which is called repeatedly to get the
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	8 next line of input (or b"" for EOF). It generates 5-tuples with these
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	9 members:
18351 44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	10
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	11 the token type (see token.py)
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	12 the token (a string)
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	13 the starting (row, column) indices of the token (a 2-tuple of ints)
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	14 the ending (row, column) indices of the token (a 2-tuple of ints)
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	15 the original line (string)
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	16
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	17 It is designed to match the working of the Python tokenizer exactly, except
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	18 that it produces COMMENT tokens for comments and gives type OP for all
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	19 operators. Additionally, all token lists start with an ENCODING token
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	20 which tells you which encoding was used to decode the bytes stream.
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	21 """
5173 ff4f8fa8a273 Fixed doc string, added __version__, fixed 1 bug. Guido van Rossum <guido@python.org> parents: 5172 diff changeset	22
16899 320fa50a1a7d Add __author__ and __credits__ variables. Ka-Ping Yee <ping@zesty.ca> parents: 16889 diff changeset	23 __author__ = 'Ka-Ping Yee <ping@lfw.org>'
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	24 __credits__ = ('GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, '
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	25 'Skip Montanaro, Raymond Hettinger, Trent Nelson, '
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	26 'Michael Foord')
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	27 import re
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	28 import sys
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	29 from token import *
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	30 from codecs import lookup, BOM_UTF8
64581 fea86c5e3385 Experiment: Let collections.namedtuple() do the work. This should work now that _collections is pre-built. The buildbots will tell us shortly. Raymond Hettinger <python@rcn.com> parents: 64577 diff changeset	31 import collections
66097 306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	32 from io import TextIOWrapper
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	33 cookie_re = re.compile("coding[:=]\s*([-\w.]+)")
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	34
16889 0feb63490093 final round of __all__ lists (I hope) - skipped urllib2 because Moshe may be Skip Montanaro <skip@pobox.com> parents: 16514 diff changeset	35 import token
66148 98bd3ca1ed86 Issue #10386: Added __all__ to token module; this simplifies importing Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66097 diff changeset	36 __all__ = token.__all__ + ["COMMENT", "tokenize", "detect_encoding",
98bd3ca1ed86 Issue #10386: Added __all__ to token module; this simplifies importing Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66097 diff changeset	37 "NL", "untokenize", "ENCODING", "TokenInfo"]
16889 0feb63490093 final round of __all__ lists (I hope) - skipped urllib2 because Moshe may be Skip Montanaro <skip@pobox.com> parents: 16514 diff changeset	38 del token
0feb63490093 final round of __all__ lists (I hope) - skipped urllib2 because Moshe may be Skip Montanaro <skip@pobox.com> parents: 16514 diff changeset	39
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	40 COMMENT = N_TOKENS
1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	41 tok_name[COMMENT] = 'COMMENT'
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	42 NL = N_TOKENS + 1
d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	43 tok_name[NL] = 'NL'
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	44 ENCODING = N_TOKENS + 2
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	45 tok_name[ENCODING] = 'ENCODING'
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	46 N_TOKENS += 3
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	47
64581 fea86c5e3385 Experiment: Let collections.namedtuple() do the work. This should work now that _collections is pre-built. The buildbots will tell us shortly. Raymond Hettinger <python@rcn.com> parents: 64577 diff changeset	48 class TokenInfo(collections.namedtuple('TokenInfo', 'type string start end line')):
53597 5f81c45bdf6d Remove dependency on the collections module. Raymond Hettinger <python@rcn.com> parents: 53584 diff changeset	49 def __repr__(self):
64582 71b264e1bb71 A little bit more readable repr method. Raymond Hettinger <python@rcn.com> parents: 64581 diff changeset	50 annotated_type = '%d (%s)' % (self.type, tok_name[self.type])
71b264e1bb71 A little bit more readable repr method. Raymond Hettinger <python@rcn.com> parents: 64581 diff changeset	51 return ('TokenInfo(type=%s, string=%r, start=%r, end=%r, line=%r)' %
71b264e1bb71 A little bit more readable repr method. Raymond Hettinger <python@rcn.com> parents: 64581 diff changeset	52 self._replace(type=annotated_type))
53597 5f81c45bdf6d Remove dependency on the collections module. Raymond Hettinger <python@rcn.com> parents: 53584 diff changeset	53
16514 90c600a819e9 String method conversion. Eric S. Raymond <esr@thyrsus.com> parents: 15898 diff changeset	54 def group(*choices): return '(' + '\|'.join(choices) + ')'
28089 14f65d661084 Get rid of many apply() calls. Guido van Rossum <guido@python.org> parents: 26032 diff changeset	55 def any(choices): return group(choices) + '*'
14f65d661084 Get rid of many apply() calls. Guido van Rossum <guido@python.org> parents: 26032 diff changeset	56 def maybe(choices): return group(choices) + '?'
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	57
48769 83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	58 # Note: we use unicode matching for names ("\w") but ascii matching for
83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	59 # number literals.
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	60 Whitespace = r'[ \f\t]*'
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	61 Comment = r'#[^\r\n]*'
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	62 Ignore = Whitespace + any(r'\\\r?\n' + Whitespace) + maybe(Comment)
64317 fe7214bc6d66 handle names starting with non-ascii characters correctly #9712 Benjamin Peterson <benjamin@python.org> parents: 59701 diff changeset	63 Name = r'\w+'
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	64
48769 83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	65 Hexnumber = r'0[xX][0-9a-fA-F]+'
44782 ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from Georg Brandl <georg@python.org> parents: 43833 diff changeset	66 Binnumber = r'0[bB][01]+'
ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from Georg Brandl <georg@python.org> parents: 43833 diff changeset	67 Octnumber = r'0[oO][0-7]+'
48769 83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	68 Decnumber = r'(?:0+\|[1-9][0-9]*)'
42084 229c28cb9afb Merged revisions 55817-55961 via svnmerge from Guido van Rossum <guido@python.org> parents: 41717 diff changeset	69 Intnumber = group(Hexnumber, Binnumber, Octnumber, Decnumber)
48769 83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	70 Exponent = r'[eE][-+]?[0-9]+'
83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	71 Pointfloat = group(r'[0-9]+\.[0-9]*', r'\.[0-9]+') + maybe(Exponent)
83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	72 Expfloat = r'[0-9]+' + Exponent
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	73 Floatnumber = group(Pointfloat, Expfloat)
48769 83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	74 Imagnumber = group(r'[0-9]+[jJ]', Floatnumber + r'[jJ]')
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	75 Number = group(Imagnumber, Floatnumber, Intnumber)
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	76
15080 0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	77 # Tail end of ' string.
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	78 Single = r"[^'\\](?:\\.[^'\\])*'"
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	79 # Tail end of " string.
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	80 Double = r'[^"\\](?:\\.[^"\\])*"'
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	81 # Tail end of ''' string.
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	82 Single3 = r"[^'\\](?:(?:\\.\|'(?!''))[^'\\])*'''"
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	83 # Tail end of """ string.
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	84 Double3 = r'[^"\\](?:(?:\\.\|"(?!""))[^"\\])*"""'
43833 ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	85 Triple = group("[bB]?[rR]?'''", '[bB]?[rR]?"""')
15080 0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	86 # Single-line ' or " string.
43833 ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	87 String = group(r"[bB]?[rR]?'[^\n'\\](?:\\.[^\n'\\])*'",
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	88 r'[bB]?[rR]?"[^\n"\\](?:\\.[^\n"\\])*"')
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	89
15080 0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	90 # Because of leftmost-then-longest match semantics, be sure to put the
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	91 # longest operators first (e.g., if = came before ==, == would get
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	92 # recognized as two instances of =).
39648 efe3f2c0da5e Killed the <> operator. You must now use !=. Guido van Rossum <guido@python.org> parents: 39610 diff changeset	93 Operator = group(r"\\=?", r">>=?", r"<<=?", r"!=",
40529 54cd47788811 PEP 3107 - Function Annotations thanks to Tony Lownds Neal Norwitz <nnorwitz@gmail.com> parents: 40450 diff changeset	94 r"//=?", r"->",
15080 0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	95 r"[+\-*/%&\|^=<>]=?",
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	96 r"~")
14166 e9c2f830f9ba Update for augmented assignment, tested & approved by Guido. Thomas Wouters <thomas@python.org> parents: 14021 diff changeset	97
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	98 Bracket = '[][(){}]'
41230 b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis. Georg Brandl <georg@python.org> parents: 40775 diff changeset	99 Special = group(r'\r?\n', r'\.\.\.', r'[:;.,@]')
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	100 Funny = group(Operator, Bracket, Special)
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	101
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	102 PlainToken = group(Number, Funny, String, Name)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	103 Token = Ignore + PlainToken
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	104
15080 0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	105 # First (or only) line of ' or " string.
43833 ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	106 ContStr = group(r"[bB]?[rR]?'[^\n'\\](?:\\.[^\n'\\])*" +
15898 8366fcfb6090 Add tokenizer support and tests for u'', U"", uR'', Ur"", etc. Ka-Ping Yee <ping@zesty.ca> parents: 15873 diff changeset	107 group("'", r'\\\r?\n'),
43833 ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	108 r'[bB]?[rR]?"[^\n"\\](?:\\.[^\n"\\])*' +
15898 8366fcfb6090 Add tokenizer support and tests for u'', U"", uR'', Ur"", etc. Ka-Ping Yee <ping@zesty.ca> parents: 15873 diff changeset	109 group('"', r'\\\r?\n'))
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	110 PseudoExtras = group(r'\\\r?\n', Comment, Triple)
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	111 PseudoToken = Whitespace + group(PseudoExtras, Number, Funny, ContStr, Name)
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	112
64317 fe7214bc6d66 handle names starting with non-ascii characters correctly #9712 Benjamin Peterson <benjamin@python.org> parents: 59701 diff changeset	113 def _compile(expr):
fe7214bc6d66 handle names starting with non-ascii characters correctly #9712 Benjamin Peterson <benjamin@python.org> parents: 59701 diff changeset	114 return re.compile(expr, re.UNICODE)
fe7214bc6d66 handle names starting with non-ascii characters correctly #9712 Benjamin Peterson <benjamin@python.org> parents: 59701 diff changeset	115
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	116 tokenprog, pseudoprog, single3prog, double3prog = map(
64317 fe7214bc6d66 handle names starting with non-ascii characters correctly #9712 Benjamin Peterson <benjamin@python.org> parents: 59701 diff changeset	117 _compile, (Token, PseudoToken, Single3, Double3))
fe7214bc6d66 handle names starting with non-ascii characters correctly #9712 Benjamin Peterson <benjamin@python.org> parents: 59701 diff changeset	118 endprogs = {"'": _compile(Single), '"': _compile(Double),
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	119 "'''": single3prog, '"""': double3prog,
6532 d369629e8c27 New, fixed version with proper r"..." and R"..." support from Ka-Ping. Guido van Rossum <guido@python.org> parents: 6531 diff changeset	120 "r'''": single3prog, 'r"""': double3prog,
43833 ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	121 "b'''": single3prog, 'b"""': double3prog,
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	122 "br'''": single3prog, 'br"""': double3prog,
15898 8366fcfb6090 Add tokenizer support and tests for u'', U"", uR'', Ur"", etc. Ka-Ping Yee <ping@zesty.ca> parents: 15873 diff changeset	123 "R'''": single3prog, 'R"""': double3prog,
43833 ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	124 "B'''": single3prog, 'B"""': double3prog,
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	125 "bR'''": single3prog, 'bR"""': double3prog,
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	126 "Br'''": single3prog, 'Br"""': double3prog,
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	127 "BR'''": single3prog, 'BR"""': double3prog,
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	128 'r': None, 'R': None, 'b': None, 'B': None}
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	129
25016 0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	130 triple_quoted = {}
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	131 for t in ("'''", '"""',
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	132 "r'''", 'r"""', "R'''", 'R"""',
43833 ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	133 "b'''", 'b"""', "B'''", 'B"""',
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	134 "br'''", 'br"""', "Br'''", 'Br"""',
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	135 "bR'''", 'bR"""', "BR'''", 'BR"""'):
25016 0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	136 triple_quoted[t] = t
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	137 single_quoted = {}
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	138 for t in ("'", '"',
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	139 "r'", 'r"', "R'", 'R"',
43833 ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	140 "b'", 'b"', "B'", 'B"',
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	141 "br'", 'br"', "Br'", 'Br"',
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	142 "bR'", 'bR"', "BR'", 'BR"' ):
25016 0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	143 single_quoted[t] = t
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	144
64317 fe7214bc6d66 handle names starting with non-ascii characters correctly #9712 Benjamin Peterson <benjamin@python.org> parents: 59701 diff changeset	145 del _compile
fe7214bc6d66 handle names starting with non-ascii characters correctly #9712 Benjamin Peterson <benjamin@python.org> parents: 59701 diff changeset	146
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	147 tabsize = 8
14021 5b34b70ecf22 Convert some old-style string exceptions to class exceptions. Fred Drake <fdrake@acm.org> parents: 7839 diff changeset	148
17178 707f95d05e69 Provide a StopTokenizing exception for conveniently exiting the loop. Ka-Ping Yee <ping@zesty.ca> parents: 16901 diff changeset	149 class TokenError(Exception): pass
707f95d05e69 Provide a StopTokenizing exception for conveniently exiting the loop. Ka-Ping Yee <ping@zesty.ca> parents: 16901 diff changeset	150
707f95d05e69 Provide a StopTokenizing exception for conveniently exiting the loop. Ka-Ping Yee <ping@zesty.ca> parents: 16901 diff changeset	151 class StopTokenizing(Exception): pass
14021 5b34b70ecf22 Convert some old-style string exceptions to class exceptions. Fred Drake <fdrake@acm.org> parents: 7839 diff changeset	152
18148 7bc473b2c654 Merging the gen-branch into the main line, at Guido's direction. Yay! Tim Peters <tim.peters@gmail.com> parents: 17178 diff changeset	153
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	154 class Untokenizer:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	155
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	156 def __init__(self):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	157 self.tokens = []
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	158 self.prev_row = 1
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	159 self.prev_col = 0
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	160 self.encoding = None
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	161
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	162 def add_whitespace(self, start):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	163 row, col = start
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	164 assert row <= self.prev_row
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	165 col_offset = col - self.prev_col
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	166 if col_offset:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	167 self.tokens.append(" " * col_offset)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	168
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	169 def untokenize(self, iterable):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	170 for t in iterable:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	171 if len(t) == 2:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	172 self.compat(t, iterable)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	173 break
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	174 tok_type, token, start, end, line = t
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	175 if tok_type == ENCODING:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	176 self.encoding = token
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	177 continue
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	178 self.add_whitespace(start)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	179 self.tokens.append(token)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	180 self.prev_row, self.prev_col = end
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	181 if tok_type in (NEWLINE, NL):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	182 self.prev_row += 1
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	183 self.prev_col = 0
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	184 return "".join(self.tokens)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	185
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	186 def compat(self, token, iterable):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	187 startline = False
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	188 indents = []
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	189 toks_append = self.tokens.append
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	190 toknum, tokval = token
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	191
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	192 if toknum in (NAME, NUMBER):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	193 tokval += ' '
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	194 if toknum in (NEWLINE, NL):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	195 startline = True
46145 da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	196 prevstring = False
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	197 for tok in iterable:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	198 toknum, tokval = tok[:2]
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	199 if toknum == ENCODING:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	200 self.encoding = tokval
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	201 continue
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	202
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	203 if toknum in (NAME, NUMBER):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	204 tokval += ' '
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	205
46145 da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	206 # Insert a space between two consecutive strings
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	207 if toknum == STRING:
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	208 if prevstring:
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	209 tokval = ' ' + tokval
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	210 prevstring = True
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	211 else:
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	212 prevstring = False
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	213
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	214 if toknum == INDENT:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	215 indents.append(tokval)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	216 continue
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	217 elif toknum == DEDENT:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	218 indents.pop()
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	219 continue
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	220 elif toknum in (NEWLINE, NL):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	221 startline = True
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	222 elif startline and indents:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	223 toks_append(indents[-1])
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	224 startline = False
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	225 toks_append(tokval)
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	226
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	227
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	228 def untokenize(iterable):
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	229 """Transform tokens back into Python source code.
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	230 It returns a bytes object, encoded using the ENCODING
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	231 token, which is the first token sequence output by tokenize.
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	232
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	233 Each element returned by the iterable must be a token sequence
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	234 with at least two elements, a token number and token value. If
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	235 only two tokens are passed, the resulting output is poor.
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	236
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	237 Round-trip invariant for full input:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	238 Untokenized source will match input source exactly
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	239
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	240 Round-trip invariant for limited intput:
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	241 # Output bytes will tokenize the back to the input
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	242 t1 = [tok[:2] for tok in tokenize(f.readline)]
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	243 newcode = untokenize(t1)
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	244 readline = BytesIO(newcode).readline
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	245 t2 = [tok[:2] for tok in tokenize(readline)]
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	246 assert t1 == t2
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	247 """
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	248 ut = Untokenizer()
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	249 out = ut.untokenize(iterable)
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	250 if ut.encoding is not None:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	251 out = out.encode(ut.encoding)
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	252 return out
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	253
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	254
56434 12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	255 def _get_normal_name(orig_enc):
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	256 """Imitates get_normal_name in tokenizer.c."""
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	257 # Only care about the first 12 characters.
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	258 enc = orig_enc[:12].lower().replace("_", "-")
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	259 if enc == "utf-8" or enc.startswith("utf-8-"):
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	260 return "utf-8"
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	261 if enc in ("latin-1", "iso-8859-1", "iso-latin-1") or \
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	262 enc.startswith(("latin-1-", "iso-8859-1-", "iso-latin-1-")):
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	263 return "iso-8859-1"
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	264 return orig_enc
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	265
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	266 def detect_encoding(readline):
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	267 """
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	268 The detect_encoding() function is used to detect the encoding that should
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	269 be used to decode a Python source file. It requires one argment, readline,
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	270 in the same way as the tokenize() generator.
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	271
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	272 It will call readline a maximum of twice, and return the encoding used
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	273 (as a string) and a list of any lines (left as bytes) it has read in.
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	274
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	275 It detects the encoding from the presence of a utf-8 bom or an encoding
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	276 cookie as specified in pep-0263. If both a bom and a cookie are present,
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	277 but disagree, a SyntaxError will be raised. If the encoding cookie is an
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	278 invalid charset, raise a SyntaxError. Note that if a utf-8 bom is found,
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	279 'utf-8-sig' is returned.
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	280
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	281 If no encoding is specified, then the default of 'utf-8' will be returned.
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	282 """
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	283 bom_found = False
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	284 encoding = None
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	285 default = 'utf-8'
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	286 def read_or_stop():
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	287 try:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	288 return readline()
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	289 except StopIteration:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	290 return b''
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	291
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	292 def find_cookie(line):
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	293 try:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	294 line_string = line.decode('ascii')
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	295 except UnicodeDecodeError:
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	296 return None
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	297
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	298 matches = cookie_re.findall(line_string)
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	299 if not matches:
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	300 return None
56434 12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	301 encoding = _get_normal_name(matches[0])
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	302 try:
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	303 codec = lookup(encoding)
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	304 except LookupError:
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	305 # This behaviour mimics the Python interpreter
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	306 raise SyntaxError("unknown encoding: " + encoding)
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	307
59701 15b87ea1413d fix for files with coding cookies and BOMs Benjamin Peterson <benjamin@python.org> parents: 59700 diff changeset	308 if bom_found:
15b87ea1413d fix for files with coding cookies and BOMs Benjamin Peterson <benjamin@python.org> parents: 59700 diff changeset	309 if codec.name != 'utf-8':
15b87ea1413d fix for files with coding cookies and BOMs Benjamin Peterson <benjamin@python.org> parents: 59700 diff changeset	310 # This behaviour mimics the Python interpreter
15b87ea1413d fix for files with coding cookies and BOMs Benjamin Peterson <benjamin@python.org> parents: 59700 diff changeset	311 raise SyntaxError('encoding problem: utf-8')
15b87ea1413d fix for files with coding cookies and BOMs Benjamin Peterson <benjamin@python.org> parents: 59700 diff changeset	312 encoding += '-sig'
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	313 return encoding
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	314
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	315 first = read_or_stop()
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	316 if first.startswith(BOM_UTF8):
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	317 bom_found = True
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	318 first = first[3:]
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	319 default = 'utf-8-sig'
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	320 if not first:
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	321 return default, []
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	322
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	323 encoding = find_cookie(first)
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	324 if encoding:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	325 return encoding, [first]
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	326
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	327 second = read_or_stop()
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	328 if not second:
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	329 return default, [first]
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	330
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	331 encoding = find_cookie(second)
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	332 if encoding:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	333 return encoding, [first, second]
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	334
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	335 return default, [first, second]
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	336
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	337
66097 306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	338 _builtin_open = open
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	339
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	340 def open(filename):
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	341 """Open a file in read only mode using the encoding detected by
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	342 detect_encoding().
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	343 """
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	344 buffer = _builtin_open(filename, 'rb')
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	345 encoding, lines = detect_encoding(buffer.readline)
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	346 buffer.seek(0)
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	347 text = TextIOWrapper(buffer, encoding, line_buffering=True)
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	348 text.mode = 'r'
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	349 return text
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	350
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	351
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	352 def tokenize(readline):
23455 68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	353 """
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	354 The tokenize() generator requires one argment, readline, which
23455 68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	355 must be a callable object which provides the same interface as the
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	356 readline() method of built-in file objects. Each call to the function
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	357 should return one line of input as bytes. Alternately, readline
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	358 can be a callable function terminating with StopIteration:
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	359 readline = open(myfile, 'rb').__next__ # Example of alternate readline
23521 a90faa5f9417 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 23455 diff changeset	360
23455 68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	361 The generator produces 5-tuples with these members: the token type; the
68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	362 token string; a 2-tuple (srow, scol) of ints specifying the row and
68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	363 column where the token begins in the source; a 2-tuple (erow, ecol) of
68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	364 ints specifying the row and column where the token ends in the source;
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	365 and the line on which the token was found. The line passed is the
23521 a90faa5f9417 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 23455 diff changeset	366 logical line; continuation lines are included.
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	367
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	368 The first token sequence will always be an ENCODING token
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	369 which tells you which encoding was used to decode the bytes stream.
23455 68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	370 """
57235 555045541794 simply by using itertools.chain() Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	371 # This import is here to avoid problems when the itertools module is not
555045541794 simply by using itertools.chain() Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	372 # built yet and tokenize is imported.
57237 94f2705def5c use some more itertools magic to make '' be yielded after readline is done Benjamin Peterson <benjamin@python.org> parents: 57235 diff changeset	373 from itertools import chain, repeat
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	374 encoding, consumed = detect_encoding(readline)
57237 94f2705def5c use some more itertools magic to make '' be yielded after readline is done Benjamin Peterson <benjamin@python.org> parents: 57235 diff changeset	375 rl_gen = iter(readline, b"")
94f2705def5c use some more itertools magic to make '' be yielded after readline is done Benjamin Peterson <benjamin@python.org> parents: 57235 diff changeset	376 empty = repeat(b"")
94f2705def5c use some more itertools magic to make '' be yielded after readline is done Benjamin Peterson <benjamin@python.org> parents: 57235 diff changeset	377 return _tokenize(chain(consumed, rl_gen, empty).__next__, encoding)
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	378
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	379
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	380 def _tokenize(readline, encoding):
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	381 lnum = parenlev = continued = 0
64317 fe7214bc6d66 handle names starting with non-ascii characters correctly #9712 Benjamin Peterson <benjamin@python.org> parents: 59701 diff changeset	382 numchars = '0123456789'
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	383 contstr, needcont = '', 0
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	384 contline = None
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	385 indents = [0]
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	386
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	387 if encoding is not None:
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	388 if encoding == "utf-8-sig":
72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	389 # BOM will already have been stripped.
72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	390 encoding = "utf-8"
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	391 yield TokenInfo(ENCODING, encoding, (0, 0), (0, 0), '')
47481 0d7ccb50007c use the more idomatic (and Py3k faster) while True Benjamin Peterson <benjamin@python.org> parents: 46145 diff changeset	392 while True: # loop over lines in stream
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	393 try:
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	394 line = readline()
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	395 except StopIteration:
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	396 line = b''
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	397
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	398 if encoding is not None:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	399 line = line.decode(encoding)
57210 a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	400 lnum += 1
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	401 pos, max = 0, len(line)
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	402
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	403 if contstr: # continued string
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	404 if not line:
43020 907a45df1295 Raise statement normalization in Lib/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	405 raise TokenError("EOF in multi-line string", strstart)
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	406 endmatch = endprog.match(line)
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	407 if endmatch:
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	408 pos = end = endmatch.end(0)
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	409 yield TokenInfo(STRING, contstr + line[:end],
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	410 strstart, (lnum, end), contline + line)
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	411 contstr, needcont = '', 0
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	412 contline = None
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	413 elif needcont and line[-2:] != '\\\n' and line[-3:] != '\\\r\n':
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	414 yield TokenInfo(ERRORTOKEN, contstr + line,
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	415 strstart, (lnum, len(line)), contline)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	416 contstr = ''
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	417 contline = None
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	418 continue
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	419 else:
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	420 contstr = contstr + line
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	421 contline = contline + line
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	422 continue
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	423
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	424 elif parenlev == 0 and not continued: # new statement
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	425 if not line: break
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	426 column = 0
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	427 while pos < max: # measure leading whitespace
57210 a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	428 if line[pos] == ' ':
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	429 column += 1
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	430 elif line[pos] == '\t':
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	431 column = (column//tabsize + 1)*tabsize
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	432 elif line[pos] == '\f':
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	433 column = 0
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	434 else:
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	435 break
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	436 pos += 1
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	437 if pos == max:
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	438 break
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	439
1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	440 if line[pos] in '#\r\n': # skip comments or blank lines
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	441 if line[pos] == '#':
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	442 comment_token = line[pos:].rstrip('\r\n')
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	443 nl_pos = pos + len(comment_token)
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	444 yield TokenInfo(COMMENT, comment_token,
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	445 (lnum, pos), (lnum, pos + len(comment_token)), line)
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	446 yield TokenInfo(NL, line[nl_pos:],
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	447 (lnum, nl_pos), (lnum, len(line)), line)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	448 else:
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	449 yield TokenInfo((NL, COMMENT)[line[pos] == '#'], line[pos:],
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	450 (lnum, pos), (lnum, len(line)), line)
1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	451 continue
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	452
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	453 if column > indents[-1]: # count indents or dedents
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	454 indents.append(column)
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	455 yield TokenInfo(INDENT, line[:pos], (lnum, 0), (lnum, pos), line)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	456 while column < indents[-1]:
35119 04868959f1fe SF bug #1224621: tokenize module does not detect inconsistent dedents Raymond Hettinger <python@rcn.com> parents: 35058 diff changeset	457 if column not in indents:
04868959f1fe SF bug #1224621: tokenize module does not detect inconsistent dedents Raymond Hettinger <python@rcn.com> parents: 35058 diff changeset	458 raise IndentationError(
39610 3674f20ac781 Merge current trunk into p3yk. This includes the PyNumber_Index API change, Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	459 "unindent does not match any outer indentation level",
3674f20ac781 Merge current trunk into p3yk. This includes the PyNumber_Index API change, Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	460 ("<tokenize>", lnum, pos, line))
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	461 indents = indents[:-1]
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	462 yield TokenInfo(DEDENT, '', (lnum, pos), (lnum, pos), line)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	463
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	464 else: # continued statement
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	465 if not line:
43020 907a45df1295 Raise statement normalization in Lib/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	466 raise TokenError("EOF in multi-line statement", (lnum, 0))
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	467 continued = 0
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	468
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	469 while pos < max:
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	470 pseudomatch = pseudoprog.match(line, pos)
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	471 if pseudomatch: # scan for tokens
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	472 start, end = pseudomatch.span(1)
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	473 spos, epos, pos = (lnum, start), (lnum, end), end
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	474 token, initial = line[start:end], line[start]
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	475
41230 b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis. Georg Brandl <georg@python.org> parents: 40775 diff changeset	476 if (initial in numchars or # ordinary number
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis. Georg Brandl <georg@python.org> parents: 40775 diff changeset	477 (initial == '.' and token != '.' and token != '...')):
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	478 yield TokenInfo(NUMBER, token, spos, epos, line)
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	479 elif initial in '\r\n':
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	480 yield TokenInfo(NL if parenlev > 0 else NEWLINE,
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	481 token, spos, epos, line)
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	482 elif initial == '#':
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	483 assert not token.endswith("\n")
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	484 yield TokenInfo(COMMENT, token, spos, epos, line)
25016 0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	485 elif token in triple_quoted:
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	486 endprog = endprogs[token]
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	487 endmatch = endprog.match(line, pos)
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	488 if endmatch: # all on one line
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	489 pos = endmatch.end(0)
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	490 token = line[start:pos]
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	491 yield TokenInfo(STRING, token, spos, (lnum, pos), line)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	492 else:
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	493 strstart = (lnum, start) # multiple lines
1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	494 contstr = line[start:]
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	495 contline = line
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	496 break
25016 0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	497 elif initial in single_quoted or \
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	498 token[:2] in single_quoted or \
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	499 token[:3] in single_quoted:
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	500 if token[-1] == '\n': # continued string
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	501 strstart = (lnum, start)
15898 8366fcfb6090 Add tokenizer support and tests for u'', U"", uR'', Ur"", etc. Ka-Ping Yee <ping@zesty.ca> parents: 15873 diff changeset	502 endprog = (endprogs[initial] or endprogs[token[1]] or
8366fcfb6090 Add tokenizer support and tests for u'', U"", uR'', Ur"", etc. Ka-Ping Yee <ping@zesty.ca> parents: 15873 diff changeset	503 endprogs[token[2]])
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	504 contstr, needcont = line[start:], 1
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	505 contline = line
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	506 break
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	507 else: # ordinary string
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	508 yield TokenInfo(STRING, token, spos, epos, line)
64317 fe7214bc6d66 handle names starting with non-ascii characters correctly #9712 Benjamin Peterson <benjamin@python.org> parents: 59701 diff changeset	509 elif initial.isidentifier(): # ordinary name
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	510 yield TokenInfo(NAME, token, spos, epos, line)
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	511 elif initial == '\\': # continued stmt
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	512 continued = 1
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	513 else:
57210 a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	514 if initial in '([{':
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	515 parenlev += 1
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	516 elif initial in ')]}':
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	517 parenlev -= 1
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	518 yield TokenInfo(OP, token, spos, epos, line)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	519 else:
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	520 yield TokenInfo(ERRORTOKEN, line[pos],
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	521 (lnum, pos), (lnum, pos+1), line)
57210 a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	522 pos += 1
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	523
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	524 for indent in indents[1:]: # pop remaining indent levels
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	525 yield TokenInfo(DEDENT, '', (lnum, 0), (lnum, 0), '')
7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	526 yield TokenInfo(ENDMARKER, '', (lnum, 0), (lnum, 0), '')
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	527
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	528
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	529 # An undocumented, backwards compatible, API for all the places in the standard
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	530 # library that expect to be able to use tokenize with strings
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	531 def generate_tokens(readline):
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	532 return _tokenize(readline, None)
64577 2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	533
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	534 if __name__ == "__main__":
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	535 # Quick sanity check
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	536 s = b'''def parseline(self, line):
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	537 """Parse the line into a command name and a string containing
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	538 the arguments. Returns a tuple containing (command, args, line).
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	539 'command' and 'args' may be None if the line couldn't be parsed.
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	540 """
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	541 line = line.strip()
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	542 if not line:
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	543 return None, None, line
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	544 elif line[0] == '?':
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	545 line = 'help ' + line[1:]
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	546 elif line[0] == '!':
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	547 if hasattr(self, 'do_shell'):
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	548 line = 'shell ' + line[1:]
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	549 else:
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	550 return None, None, line
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	551 i, n = 0, len(line)
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	552 while i < n and line[i] in self.identchars: i = i+1
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	553 cmd, arg = line[:i], line[i:].strip()
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	554 return cmd, arg, line
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	555 '''
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	556 for tok in tokenize(iter(s.splitlines()).__next__):
2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	557 print(tok)

Mercurial > cpython-lru_cache-weakref

annotate Lib/tokenize.py @ 66807:d28126ebd60d