cpython-lru_cache-weakref: Lib/tokenize.py annotate

annotate Lib/tokenize.py @ 87690:66c1c9f32567 default tip

Add use_weakref option to functools.lru_cache The use_weakref option makes lru_cache create weak references to the function arguments when creating the cache key, which prevents them from staying in the memory forever even after all other references to them are gone. This is a proof of concept for http://bugs.python.org/issue19859

author	Radomir Dopieralski <devel@sheep.art.pl>
date	Tue, 03 Dec 2013 16:02:48 +0100
parents	0d5da548b80a
children

rev	line source
5173 ff4f8fa8a273 Fixed doc string, added __version__, fixed 1 bug. Guido van Rossum <guido@python.org> parents: 5172 diff changeset	1 """Tokenization help for Python programs.
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	2
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	3 tokenize(readline) is a generator that breaks a stream of bytes into
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	4 Python tokens. It decodes the bytes according to PEP-0263 for
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	5 determining source file encoding.
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	6
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	7 It accepts a readline-like method which is called repeatedly to get the
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	8 next line of input (or b"" for EOF). It generates 5-tuples with these
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	9 members:
18351 44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	10
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	11 the token type (see token.py)
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	12 the token (a string)
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	13 the starting (row, column) indices of the token (a 2-tuple of ints)
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	14 the ending (row, column) indices of the token (a 2-tuple of ints)
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	15 the original line (string)
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	16
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	17 It is designed to match the working of the Python tokenizer exactly, except
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	18 that it produces COMMENT tokens for comments and gives type OP for all
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	19 operators. Additionally, all token lists start with an ENCODING token
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	20 which tells you which encoding was used to decode the bytes stream.
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	21 """
5173 ff4f8fa8a273 Fixed doc string, added __version__, fixed 1 bug. Guido van Rossum <guido@python.org> parents: 5172 diff changeset	22
16899 320fa50a1a7d Add __author__ and __credits__ variables. Ka-Ping Yee <ping@zesty.ca> parents: 16889 diff changeset	23 __author__ = 'Ka-Ping Yee <ping@lfw.org>'
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	24 __credits__ = ('GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, '
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	25 'Skip Montanaro, Raymond Hettinger, Trent Nelson, '
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	26 'Michael Foord')
68096 ea260d393cde Issue #11074: Make 'tokenize' so it can be reloaded. Brett Cannon <bcannon@gmail.com> parents: 66148 diff changeset	27 import builtins
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	28 import re
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	29 import sys
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	30 from token import *
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	31 from codecs import lookup, BOM_UTF8
64581 fea86c5e3385 Experiment: Let collections.namedtuple() do the work. This should work now that _collections is pre-built. The buildbots will tell us shortly. Raymond Hettinger <python@rcn.com> parents: 64577 diff changeset	32 import collections
66097 306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	33 from io import TextIOWrapper
85731 2dfe8262093c Issue #18873: The tokenize module, IDLE, 2to3, and the findnocoding.py script Serhiy Storchaka <storchaka@gmail.com> parents: 80192 diff changeset	34 cookie_re = re.compile(r'^[ \t\f]#.coding[:=][ \t]*([-\w.]+)', re.ASCII)
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	35
16889 0feb63490093 final round of __all__ lists (I hope) - skipped urllib2 because Moshe may be Skip Montanaro <skip@pobox.com> parents: 16514 diff changeset	36 import token
66148 98bd3ca1ed86 Issue #10386: Added __all__ to token module; this simplifies importing Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66097 diff changeset	37 __all__ = token.__all__ + ["COMMENT", "tokenize", "detect_encoding",
98bd3ca1ed86 Issue #10386: Added __all__ to token module; this simplifies importing Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66097 diff changeset	38 "NL", "untokenize", "ENCODING", "TokenInfo"]
16889 0feb63490093 final round of __all__ lists (I hope) - skipped urllib2 because Moshe may be Skip Montanaro <skip@pobox.com> parents: 16514 diff changeset	39 del token
0feb63490093 final round of __all__ lists (I hope) - skipped urllib2 because Moshe may be Skip Montanaro <skip@pobox.com> parents: 16514 diff changeset	40
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	41 COMMENT = N_TOKENS
1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	42 tok_name[COMMENT] = 'COMMENT'
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	43 NL = N_TOKENS + 1
d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	44 tok_name[NL] = 'NL'
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	45 ENCODING = N_TOKENS + 2
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	46 tok_name[ENCODING] = 'ENCODING'
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	47 N_TOKENS += 3
74519 f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	48 EXACT_TOKEN_TYPES = {
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	49 '(': LPAR,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	50 ')': RPAR,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	51 '[': LSQB,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	52 ']': RSQB,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	53 ':': COLON,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	54 ',': COMMA,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	55 ';': SEMI,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	56 '+': PLUS,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	57 '-': MINUS,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	58 '*': STAR,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	59 '/': SLASH,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	60 '\|': VBAR,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	61 '&': AMPER,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	62 '<': LESS,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	63 '>': GREATER,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	64 '=': EQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	65 '.': DOT,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	66 '%': PERCENT,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	67 '{': LBRACE,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	68 '}': RBRACE,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	69 '==': EQEQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	70 '!=': NOTEQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	71 '<=': LESSEQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	72 '>=': GREATEREQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	73 '~': TILDE,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	74 '^': CIRCUMFLEX,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	75 '<<': LEFTSHIFT,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	76 '>>': RIGHTSHIFT,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	77 '**': DOUBLESTAR,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	78 '+=': PLUSEQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	79 '-=': MINEQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	80 '*=': STAREQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	81 '/=': SLASHEQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	82 '%=': PERCENTEQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	83 '&=': AMPEREQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	84 '\|=': VBAREQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	85 '^=': CIRCUMFLEXEQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	86 '<<=': LEFTSHIFTEQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	87 '>>=': RIGHTSHIFTEQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	88 '**=': DOUBLESTAREQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	89 '//': DOUBLESLASH,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	90 '//=': DOUBLESLASHEQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	91 '@': AT
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	92 }
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	93
64581 fea86c5e3385 Experiment: Let collections.namedtuple() do the work. This should work now that _collections is pre-built. The buildbots will tell us shortly. Raymond Hettinger <python@rcn.com> parents: 64577 diff changeset	94 class TokenInfo(collections.namedtuple('TokenInfo', 'type string start end line')):
53597 5f81c45bdf6d Remove dependency on the collections module. Raymond Hettinger <python@rcn.com> parents: 53584 diff changeset	95 def __repr__(self):
64582 71b264e1bb71 A little bit more readable repr method. Raymond Hettinger <python@rcn.com> parents: 64581 diff changeset	96 annotated_type = '%d (%s)' % (self.type, tok_name[self.type])
71b264e1bb71 A little bit more readable repr method. Raymond Hettinger <python@rcn.com> parents: 64581 diff changeset	97 return ('TokenInfo(type=%s, string=%r, start=%r, end=%r, line=%r)' %
71b264e1bb71 A little bit more readable repr method. Raymond Hettinger <python@rcn.com> parents: 64581 diff changeset	98 self._replace(type=annotated_type))
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	99
74519 f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	100 @property
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	101 def exact_type(self):
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	102 if self.type == OP and self.string in EXACT_TOKEN_TYPES:
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	103 return EXACT_TOKEN_TYPES[self.string]
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	104 else:
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	105 return self.type
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	106
16514 90c600a819e9 String method conversion. Eric S. Raymond <esr@thyrsus.com> parents: 15898 diff changeset	107 def group(*choices): return '(' + '\|'.join(choices) + ')'
28089 14f65d661084 Get rid of many apply() calls. Guido van Rossum <guido@python.org> parents: 26032 diff changeset	108 def any(choices): return group(choices) + '*'
14f65d661084 Get rid of many apply() calls. Guido van Rossum <guido@python.org> parents: 26032 diff changeset	109 def maybe(choices): return group(choices) + '?'
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	110
48769 83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	111 # Note: we use unicode matching for names ("\w") but ascii matching for
83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	112 # number literals.
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	113 Whitespace = r'[ \f\t]*'
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	114 Comment = r'#[^\r\n]*'
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	115 Ignore = Whitespace + any(r'\\\r?\n' + Whitespace) + maybe(Comment)
64318 a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	116 Name = r'\w+'
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	117
48769 83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	118 Hexnumber = r'0[xX][0-9a-fA-F]+'
44782 ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from Georg Brandl <georg@python.org> parents: 43833 diff changeset	119 Binnumber = r'0[bB][01]+'
ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from Georg Brandl <georg@python.org> parents: 43833 diff changeset	120 Octnumber = r'0[oO][0-7]+'
48769 83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	121 Decnumber = r'(?:0+\|[1-9][0-9]*)'
42084 229c28cb9afb Merged revisions 55817-55961 via svnmerge from Guido van Rossum <guido@python.org> parents: 41717 diff changeset	122 Intnumber = group(Hexnumber, Binnumber, Octnumber, Decnumber)
48769 83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	123 Exponent = r'[eE][-+]?[0-9]+'
83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	124 Pointfloat = group(r'[0-9]+\.[0-9]*', r'\.[0-9]+') + maybe(Exponent)
83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	125 Expfloat = r'[0-9]+' + Exponent
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	126 Floatnumber = group(Pointfloat, Expfloat)
48769 83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	127 Imagnumber = group(r'[0-9]+[jJ]', Floatnumber + r'[jJ]')
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	128 Number = group(Imagnumber, Floatnumber, Intnumber)
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	129
77522 8e47e9af826e Issue #15096: Drop support for the ur string prefix Christian Heimes <c.heimes@cheimes.de> parents: 77475 diff changeset	130 StringPrefix = r'(?:[bB][rR]?\|[rR][bB]?\|[uU])?'
75383 2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	131
15080 0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	132 # Tail end of ' string.
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	133 Single = r"[^'\\](?:\\.[^'\\])*'"
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	134 # Tail end of " string.
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	135 Double = r'[^"\\](?:\\.[^"\\])*"'
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	136 # Tail end of ''' string.
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	137 Single3 = r"[^'\\](?:(?:\\.\|'(?!''))[^'\\])*'''"
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	138 # Tail end of """ string.
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	139 Double3 = r'[^"\\](?:(?:\\.\|"(?!""))[^"\\])*"""'
75383 2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	140 Triple = group(StringPrefix + "'''", StringPrefix + '"""')
15080 0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	141 # Single-line ' or " string.
75383 2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	142 String = group(StringPrefix + r"'[^\n'\\](?:\\.[^\n'\\])*'",
2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	143 StringPrefix + r'"[^\n"\\](?:\\.[^\n"\\])*"')
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	144
15080 0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	145 # Because of leftmost-then-longest match semantics, be sure to put the
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	146 # longest operators first (e.g., if = came before ==, == would get
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	147 # recognized as two instances of =).
39648 efe3f2c0da5e Killed the <> operator. You must now use !=. Guido van Rossum <guido@python.org> parents: 39495 diff changeset	148 Operator = group(r"\\=?", r">>=?", r"<<=?", r"!=",
40529 54cd47788811 PEP 3107 - Function Annotations thanks to Tony Lownds Neal Norwitz <nnorwitz@gmail.com> parents: 40450 diff changeset	149 r"//=?", r"->",
15080 0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	150 r"[+\-*/%&\|^=<>]=?",
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	151 r"~")
14166 e9c2f830f9ba Update for augmented assignment, tested & approved by Guido. Thomas Wouters <thomas@python.org> parents: 14021 diff changeset	152
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	153 Bracket = '[][(){}]'
41230 b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis. Georg Brandl <georg@python.org> parents: 40775 diff changeset	154 Special = group(r'\r?\n', r'\.\.\.', r'[:;.,@]')
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	155 Funny = group(Operator, Bracket, Special)
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	156
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	157 PlainToken = group(Number, Funny, String, Name)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	158 Token = Ignore + PlainToken
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	159
15080 0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	160 # First (or only) line of ' or " string.
75383 2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	161 ContStr = group(StringPrefix + r"'[^\n'\\](?:\\.[^\n'\\])*" +
15898 8366fcfb6090 Add tokenizer support and tests for u'', U"", uR'', Ur"", etc. Ka-Ping Yee <ping@zesty.ca> parents: 15873 diff changeset	162 group("'", r'\\\r?\n'),
75383 2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	163 StringPrefix + r'"[^\n"\\](?:\\.[^\n"\\])*' +
15898 8366fcfb6090 Add tokenizer support and tests for u'', U"", uR'', Ur"", etc. Ka-Ping Yee <ping@zesty.ca> parents: 15873 diff changeset	164 group('"', r'\\\r?\n'))
80191 3ffff1798ed5 #16152: fix tokenize to ignore whitespace at the end of the code when no newline is found. Patch by Ned Batchelder. Ezio Melotti <ezio.melotti@gmail.com> parents: 77961 diff changeset	165 PseudoExtras = group(r'\\\r?\n\|\Z', Comment, Triple)
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	166 PseudoToken = Whitespace + group(PseudoExtras, Number, Funny, ContStr, Name)
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	167
64318 a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	168 def _compile(expr):
a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	169 return re.compile(expr, re.UNICODE)
a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	170
72853 df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	171 endpats = {"'": Single, '"': Double,
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	172 "'''": Single3, '"""': Double3,
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	173 "r'''": Single3, 'r"""': Double3,
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	174 "b'''": Single3, 'b"""': Double3,
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	175 "R'''": Single3, 'R"""': Double3,
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	176 "B'''": Single3, 'B"""': Double3,
75383 2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	177 "br'''": Single3, 'br"""': Double3,
72853 df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	178 "bR'''": Single3, 'bR"""': Double3,
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	179 "Br'''": Single3, 'Br"""': Double3,
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	180 "BR'''": Single3, 'BR"""': Double3,
75383 2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	181 "rb'''": Single3, 'rb"""': Double3,
2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	182 "Rb'''": Single3, 'Rb"""': Double3,
2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	183 "rB'''": Single3, 'rB"""': Double3,
2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	184 "RB'''": Single3, 'RB"""': Double3,
75381 ed9497d6a216 Basic support for PEP 414 without docs or tests. Armin Ronacher <armin.ronacher@active-4.com> parents: 74519 diff changeset	185 "u'''": Single3, 'u"""': Double3,
ed9497d6a216 Basic support for PEP 414 without docs or tests. Armin Ronacher <armin.ronacher@active-4.com> parents: 74519 diff changeset	186 "R'''": Single3, 'R"""': Double3,
ed9497d6a216 Basic support for PEP 414 without docs or tests. Armin Ronacher <armin.ronacher@active-4.com> parents: 74519 diff changeset	187 "U'''": Single3, 'U"""': Double3,
ed9497d6a216 Basic support for PEP 414 without docs or tests. Armin Ronacher <armin.ronacher@active-4.com> parents: 74519 diff changeset	188 'r': None, 'R': None, 'b': None, 'B': None,
ed9497d6a216 Basic support for PEP 414 without docs or tests. Armin Ronacher <armin.ronacher@active-4.com> parents: 74519 diff changeset	189 'u': None, 'U': None}
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	190
25016 0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	191 triple_quoted = {}
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	192 for t in ("'''", '"""',
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	193 "r'''", 'r"""', "R'''", 'R"""',
43833 ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	194 "b'''", 'b"""', "B'''", 'B"""',
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	195 "br'''", 'br"""', "Br'''", 'Br"""',
75381 ed9497d6a216 Basic support for PEP 414 without docs or tests. Armin Ronacher <armin.ronacher@active-4.com> parents: 74519 diff changeset	196 "bR'''", 'bR"""', "BR'''", 'BR"""',
75383 2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	197 "rb'''", 'rb"""', "rB'''", 'rB"""',
2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	198 "Rb'''", 'Rb"""', "RB'''", 'RB"""',
75381 ed9497d6a216 Basic support for PEP 414 without docs or tests. Armin Ronacher <armin.ronacher@active-4.com> parents: 74519 diff changeset	199 "u'''", 'u"""', "U'''", 'U"""',
77522 8e47e9af826e Issue #15096: Drop support for the ur string prefix Christian Heimes <c.heimes@cheimes.de> parents: 77475 diff changeset	200 ):
25016 0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	201 triple_quoted[t] = t
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	202 single_quoted = {}
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	203 for t in ("'", '"',
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	204 "r'", 'r"', "R'", 'R"',
43833 ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	205 "b'", 'b"', "B'", 'B"',
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	206 "br'", 'br"', "Br'", 'Br"',
75381 ed9497d6a216 Basic support for PEP 414 without docs or tests. Armin Ronacher <armin.ronacher@active-4.com> parents: 74519 diff changeset	207 "bR'", 'bR"', "BR'", 'BR"' ,
75383 2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	208 "rb'", 'rb"', "rB'", 'rB"',
2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	209 "Rb'", 'Rb"', "RB'", 'RB"' ,
75381 ed9497d6a216 Basic support for PEP 414 without docs or tests. Armin Ronacher <armin.ronacher@active-4.com> parents: 74519 diff changeset	210 "u'", 'u"', "U'", 'U"',
77522 8e47e9af826e Issue #15096: Drop support for the ur string prefix Christian Heimes <c.heimes@cheimes.de> parents: 77475 diff changeset	211 ):
25016 0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	212 single_quoted[t] = t
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	213
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	214 tabsize = 8
14021 5b34b70ecf22 Convert some old-style string exceptions to class exceptions. Fred Drake <fdrake@acm.org> parents: 7839 diff changeset	215
17178 707f95d05e69 Provide a StopTokenizing exception for conveniently exiting the loop. Ka-Ping Yee <ping@zesty.ca> parents: 16901 diff changeset	216 class TokenError(Exception): pass
707f95d05e69 Provide a StopTokenizing exception for conveniently exiting the loop. Ka-Ping Yee <ping@zesty.ca> parents: 16901 diff changeset	217
707f95d05e69 Provide a StopTokenizing exception for conveniently exiting the loop. Ka-Ping Yee <ping@zesty.ca> parents: 16901 diff changeset	218 class StopTokenizing(Exception): pass
14021 5b34b70ecf22 Convert some old-style string exceptions to class exceptions. Fred Drake <fdrake@acm.org> parents: 7839 diff changeset	219
18148 7bc473b2c654 Merging the gen-branch into the main line, at Guido's direction. Yay! Tim Peters <tim.peters@gmail.com> parents: 17178 diff changeset	220
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	221 class Untokenizer:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	222
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	223 def __init__(self):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	224 self.tokens = []
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	225 self.prev_row = 1
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	226 self.prev_col = 0
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	227 self.encoding = None
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	228
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	229 def add_whitespace(self, start):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	230 row, col = start
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	231 assert row <= self.prev_row
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	232 col_offset = col - self.prev_col
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	233 if col_offset:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	234 self.tokens.append(" " * col_offset)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	235
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	236 def untokenize(self, iterable):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	237 for t in iterable:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	238 if len(t) == 2:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	239 self.compat(t, iterable)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	240 break
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	241 tok_type, token, start, end, line = t
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	242 if tok_type == ENCODING:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	243 self.encoding = token
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	244 continue
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	245 self.add_whitespace(start)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	246 self.tokens.append(token)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	247 self.prev_row, self.prev_col = end
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	248 if tok_type in (NEWLINE, NL):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	249 self.prev_row += 1
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	250 self.prev_col = 0
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	251 return "".join(self.tokens)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	252
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	253 def compat(self, token, iterable):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	254 startline = False
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	255 indents = []
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	256 toks_append = self.tokens.append
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	257 toknum, tokval = token
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	258
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	259 if toknum in (NAME, NUMBER):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	260 tokval += ' '
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	261 if toknum in (NEWLINE, NL):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	262 startline = True
46145 da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	263 prevstring = False
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	264 for tok in iterable:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	265 toknum, tokval = tok[:2]
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	266 if toknum == ENCODING:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	267 self.encoding = tokval
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	268 continue
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	269
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	270 if toknum in (NAME, NUMBER):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	271 tokval += ' '
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	272
46145 da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	273 # Insert a space between two consecutive strings
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	274 if toknum == STRING:
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	275 if prevstring:
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	276 tokval = ' ' + tokval
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	277 prevstring = True
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	278 else:
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	279 prevstring = False
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	280
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	281 if toknum == INDENT:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	282 indents.append(tokval)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	283 continue
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	284 elif toknum == DEDENT:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	285 indents.pop()
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	286 continue
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	287 elif toknum in (NEWLINE, NL):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	288 startline = True
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	289 elif startline and indents:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	290 toks_append(indents[-1])
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	291 startline = False
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	292 toks_append(tokval)
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	293
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	294
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	295 def untokenize(iterable):
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	296 """Transform tokens back into Python source code.
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	297 It returns a bytes object, encoded using the ENCODING
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	298 token, which is the first token sequence output by tokenize.
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	299
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	300 Each element returned by the iterable must be a token sequence
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	301 with at least two elements, a token number and token value. If
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	302 only two tokens are passed, the resulting output is poor.
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	303
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	304 Round-trip invariant for full input:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	305 Untokenized source will match input source exactly
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	306
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	307 Round-trip invariant for limited intput:
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	308 # Output bytes will tokenize the back to the input
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	309 t1 = [tok[:2] for tok in tokenize(f.readline)]
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	310 newcode = untokenize(t1)
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	311 readline = BytesIO(newcode).readline
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	312 t2 = [tok[:2] for tok in tokenize(readline)]
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	313 assert t1 == t2
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	314 """
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	315 ut = Untokenizer()
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	316 out = ut.untokenize(iterable)
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	317 if ut.encoding is not None:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	318 out = out.encode(ut.encoding)
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	319 return out
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	320
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	321
56434 12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	322 def _get_normal_name(orig_enc):
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	323 """Imitates get_normal_name in tokenizer.c."""
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	324 # Only care about the first 12 characters.
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	325 enc = orig_enc[:12].lower().replace("_", "-")
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	326 if enc == "utf-8" or enc.startswith("utf-8-"):
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	327 return "utf-8"
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	328 if enc in ("latin-1", "iso-8859-1", "iso-latin-1") or \
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	329 enc.startswith(("latin-1-", "iso-8859-1-", "iso-latin-1-")):
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	330 return "iso-8859-1"
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	331 return orig_enc
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	332
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	333 def detect_encoding(readline):
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	334 """
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	335 The detect_encoding() function is used to detect the encoding that should
87530 3f99564b712e #19620: Fix typo in docstring (noticed by Christopher Welborn). Ezio Melotti <ezio.melotti@gmail.com> parents: 85731 diff changeset	336 be used to decode a Python source file. It requires one argument, readline,
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	337 in the same way as the tokenize() generator.
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	338
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	339 It will call readline a maximum of twice, and return the encoding used
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	340 (as a string) and a list of any lines (left as bytes) it has read in.
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	341
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	342 It detects the encoding from the presence of a utf-8 bom or an encoding
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	343 cookie as specified in pep-0263. If both a bom and a cookie are present,
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	344 but disagree, a SyntaxError will be raised. If the encoding cookie is an
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	345 invalid charset, raise a SyntaxError. Note that if a utf-8 bom is found,
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	346 'utf-8-sig' is returned.
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	347
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	348 If no encoding is specified, then the default of 'utf-8' will be returned.
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	349 """
76430 1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	350 try:
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	351 filename = readline.__self__.name
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	352 except AttributeError:
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	353 filename = None
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	354 bom_found = False
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	355 encoding = None
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	356 default = 'utf-8'
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	357 def read_or_stop():
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	358 try:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	359 return readline()
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	360 except StopIteration:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	361 return b''
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	362
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	363 def find_cookie(line):
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	364 try:
76425 b07488490001 Issue #14629: Raise SyntaxError in tokenizer.detect_encoding Martin v. Löwis <martin@v.loewis.de> parents: 68257 diff changeset	365 # Decode as UTF-8. Either the line is an encoding declaration,
b07488490001 Issue #14629: Raise SyntaxError in tokenizer.detect_encoding Martin v. Löwis <martin@v.loewis.de> parents: 68257 diff changeset	366 # in which case it should be pure ASCII, or it must be UTF-8
b07488490001 Issue #14629: Raise SyntaxError in tokenizer.detect_encoding Martin v. Löwis <martin@v.loewis.de> parents: 68257 diff changeset	367 # per default encoding.
b07488490001 Issue #14629: Raise SyntaxError in tokenizer.detect_encoding Martin v. Löwis <martin@v.loewis.de> parents: 68257 diff changeset	368 line_string = line.decode('utf-8')
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	369 except UnicodeDecodeError:
76430 1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	370 msg = "invalid or missing encoding declaration"
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	371 if filename is not None:
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	372 msg = '{} for {!r}'.format(msg, filename)
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	373 raise SyntaxError(msg)
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	374
85731 2dfe8262093c Issue #18873: The tokenize module, IDLE, 2to3, and the findnocoding.py script Serhiy Storchaka <storchaka@gmail.com> parents: 80192 diff changeset	375 match = cookie_re.match(line_string)
2dfe8262093c Issue #18873: The tokenize module, IDLE, 2to3, and the findnocoding.py script Serhiy Storchaka <storchaka@gmail.com> parents: 80192 diff changeset	376 if not match:
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	377 return None
85731 2dfe8262093c Issue #18873: The tokenize module, IDLE, 2to3, and the findnocoding.py script Serhiy Storchaka <storchaka@gmail.com> parents: 80192 diff changeset	378 encoding = _get_normal_name(match.group(1))
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	379 try:
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	380 codec = lookup(encoding)
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	381 except LookupError:
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	382 # This behaviour mimics the Python interpreter
76430 1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	383 if filename is None:
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	384 msg = "unknown encoding: " + encoding
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	385 else:
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	386 msg = "unknown encoding for {!r}: {}".format(filename,
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	387 encoding)
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	388 raise SyntaxError(msg)
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	389
59701 15b87ea1413d fix for files with coding cookies and BOMs Benjamin Peterson <benjamin@python.org> parents: 59700 diff changeset	390 if bom_found:
77961 5020afc0b7c9 Issue #14990: tokenize: correctly fail with SyntaxError on invalid encoding declaration. Florent Xicluna <florent.xicluna@gmail.com> parents: 76425 diff changeset	391 if encoding != 'utf-8':
59701 15b87ea1413d fix for files with coding cookies and BOMs Benjamin Peterson <benjamin@python.org> parents: 59700 diff changeset	392 # This behaviour mimics the Python interpreter
76430 1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	393 if filename is None:
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	394 msg = 'encoding problem: utf-8'
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	395 else:
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	396 msg = 'encoding problem for {!r}: utf-8'.format(filename)
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	397 raise SyntaxError(msg)
59701 15b87ea1413d fix for files with coding cookies and BOMs Benjamin Peterson <benjamin@python.org> parents: 59700 diff changeset	398 encoding += '-sig'
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	399 return encoding
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	400
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	401 first = read_or_stop()
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	402 if first.startswith(BOM_UTF8):
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	403 bom_found = True
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	404 first = first[3:]
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	405 default = 'utf-8-sig'
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	406 if not first:
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	407 return default, []
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	408
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	409 encoding = find_cookie(first)
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	410 if encoding:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	411 return encoding, [first]
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	412
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	413 second = read_or_stop()
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	414 if not second:
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	415 return default, [first]
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	416
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	417 encoding = find_cookie(second)
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	418 if encoding:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	419 return encoding, [first, second]
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	420
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	421 return default, [first, second]
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	422
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	423
66097 306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	424 def open(filename):
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	425 """Open a file in read only mode using the encoding detected by
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	426 detect_encoding().
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	427 """
68096 ea260d393cde Issue #11074: Make 'tokenize' so it can be reloaded. Brett Cannon <bcannon@gmail.com> parents: 66148 diff changeset	428 buffer = builtins.open(filename, 'rb')
66097 306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	429 encoding, lines = detect_encoding(buffer.readline)
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	430 buffer.seek(0)
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	431 text = TextIOWrapper(buffer, encoding, line_buffering=True)
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	432 text.mode = 'r'
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	433 return text
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	434
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	435
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	436 def tokenize(readline):
23455 68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	437 """
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	438 The tokenize() generator requires one argment, readline, which
23455 68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	439 must be a callable object which provides the same interface as the
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	440 readline() method of built-in file objects. Each call to the function
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	441 should return one line of input as bytes. Alternately, readline
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	442 can be a callable function terminating with StopIteration:
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	443 readline = open(myfile, 'rb').__next__ # Example of alternate readline
23521 a90faa5f9417 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 23455 diff changeset	444
23455 68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	445 The generator produces 5-tuples with these members: the token type; the
68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	446 token string; a 2-tuple (srow, scol) of ints specifying the row and
68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	447 column where the token begins in the source; a 2-tuple (erow, ecol) of
68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	448 ints specifying the row and column where the token ends in the source;
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	449 and the line on which the token was found. The line passed is the
23521 a90faa5f9417 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 23455 diff changeset	450 logical line; continuation lines are included.
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	451
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	452 The first token sequence will always be an ENCODING token
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	453 which tells you which encoding was used to decode the bytes stream.
23455 68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	454 """
57235 555045541794 simply by using itertools.chain() Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	455 # This import is here to avoid problems when the itertools module is not
555045541794 simply by using itertools.chain() Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	456 # built yet and tokenize is imported.
57237 94f2705def5c use some more itertools magic to make '' be yielded after readline is done Benjamin Peterson <benjamin@python.org> parents: 57235 diff changeset	457 from itertools import chain, repeat
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	458 encoding, consumed = detect_encoding(readline)
57237 94f2705def5c use some more itertools magic to make '' be yielded after readline is done Benjamin Peterson <benjamin@python.org> parents: 57235 diff changeset	459 rl_gen = iter(readline, b"")
94f2705def5c use some more itertools magic to make '' be yielded after readline is done Benjamin Peterson <benjamin@python.org> parents: 57235 diff changeset	460 empty = repeat(b"")
94f2705def5c use some more itertools magic to make '' be yielded after readline is done Benjamin Peterson <benjamin@python.org> parents: 57235 diff changeset	461 return _tokenize(chain(consumed, rl_gen, empty).__next__, encoding)
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	462
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	463
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	464 def _tokenize(readline, encoding):
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	465 lnum = parenlev = continued = 0
64318 a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	466 numchars = '0123456789'
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	467 contstr, needcont = '', 0
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	468 contline = None
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	469 indents = [0]
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	470
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	471 if encoding is not None:
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	472 if encoding == "utf-8-sig":
72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	473 # BOM will already have been stripped.
72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	474 encoding = "utf-8"
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	475 yield TokenInfo(ENCODING, encoding, (0, 0), (0, 0), '')
47481 0d7ccb50007c use the more idomatic (and Py3k faster) while True Benjamin Peterson <benjamin@python.org> parents: 46145 diff changeset	476 while True: # loop over lines in stream
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	477 try:
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	478 line = readline()
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	479 except StopIteration:
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	480 line = b''
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	481
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	482 if encoding is not None:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	483 line = line.decode(encoding)
57210 a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	484 lnum += 1
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	485 pos, max = 0, len(line)
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	486
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	487 if contstr: # continued string
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	488 if not line:
43020 907a45df1295 Raise statement normalization in Lib/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	489 raise TokenError("EOF in multi-line string", strstart)
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	490 endmatch = endprog.match(line)
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	491 if endmatch:
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	492 pos = end = endmatch.end(0)
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	493 yield TokenInfo(STRING, contstr + line[:end],
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	494 strstart, (lnum, end), contline + line)
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	495 contstr, needcont = '', 0
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	496 contline = None
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	497 elif needcont and line[-2:] != '\\\n' and line[-3:] != '\\\r\n':
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	498 yield TokenInfo(ERRORTOKEN, contstr + line,
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	499 strstart, (lnum, len(line)), contline)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	500 contstr = ''
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	501 contline = None
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	502 continue
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	503 else:
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	504 contstr = contstr + line
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	505 contline = contline + line
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	506 continue
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	507
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	508 elif parenlev == 0 and not continued: # new statement
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	509 if not line: break
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	510 column = 0
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	511 while pos < max: # measure leading whitespace
57210 a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	512 if line[pos] == ' ':
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	513 column += 1
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	514 elif line[pos] == '\t':
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	515 column = (column//tabsize + 1)*tabsize
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	516 elif line[pos] == '\f':
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	517 column = 0
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	518 else:
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	519 break
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	520 pos += 1
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	521 if pos == max:
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	522 break
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	523
1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	524 if line[pos] in '#\r\n': # skip comments or blank lines
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	525 if line[pos] == '#':
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	526 comment_token = line[pos:].rstrip('\r\n')
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	527 nl_pos = pos + len(comment_token)
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	528 yield TokenInfo(COMMENT, comment_token,
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	529 (lnum, pos), (lnum, pos + len(comment_token)), line)
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	530 yield TokenInfo(NL, line[nl_pos:],
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	531 (lnum, nl_pos), (lnum, len(line)), line)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	532 else:
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	533 yield TokenInfo((NL, COMMENT)[line[pos] == '#'], line[pos:],
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	534 (lnum, pos), (lnum, len(line)), line)
1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	535 continue
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	536
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	537 if column > indents[-1]: # count indents or dedents
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	538 indents.append(column)
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	539 yield TokenInfo(INDENT, line[:pos], (lnum, 0), (lnum, pos), line)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	540 while column < indents[-1]:
35119 04868959f1fe SF bug #1224621: tokenize module does not detect inconsistent dedents Raymond Hettinger <python@rcn.com> parents: 35058 diff changeset	541 if column not in indents:
04868959f1fe SF bug #1224621: tokenize module does not detect inconsistent dedents Raymond Hettinger <python@rcn.com> parents: 35058 diff changeset	542 raise IndentationError(
39495 14ec9cfd9760 Make tabnanny recognize IndentationErrors raised by tokenize. Georg Brandl <georg@python.org> parents: 37314 diff changeset	543 "unindent does not match any outer indentation level",
14ec9cfd9760 Make tabnanny recognize IndentationErrors raised by tokenize. Georg Brandl <georg@python.org> parents: 37314 diff changeset	544 ("<tokenize>", lnum, pos, line))
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	545 indents = indents[:-1]
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	546 yield TokenInfo(DEDENT, '', (lnum, pos), (lnum, pos), line)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	547
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	548 else: # continued statement
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	549 if not line:
43020 907a45df1295 Raise statement normalization in Lib/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	550 raise TokenError("EOF in multi-line statement", (lnum, 0))
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	551 continued = 0
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	552
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	553 while pos < max:
72853 df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	554 pseudomatch = _compile(PseudoToken).match(line, pos)
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	555 if pseudomatch: # scan for tokens
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	556 start, end = pseudomatch.span(1)
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	557 spos, epos, pos = (lnum, start), (lnum, end), end
80191 3ffff1798ed5 #16152: fix tokenize to ignore whitespace at the end of the code when no newline is found. Patch by Ned Batchelder. Ezio Melotti <ezio.melotti@gmail.com> parents: 77961 diff changeset	558 if start == end:
3ffff1798ed5 #16152: fix tokenize to ignore whitespace at the end of the code when no newline is found. Patch by Ned Batchelder. Ezio Melotti <ezio.melotti@gmail.com> parents: 77961 diff changeset	559 continue
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	560 token, initial = line[start:end], line[start]
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	561
41230 b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis. Georg Brandl <georg@python.org> parents: 40775 diff changeset	562 if (initial in numchars or # ordinary number
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis. Georg Brandl <georg@python.org> parents: 40775 diff changeset	563 (initial == '.' and token != '.' and token != '...')):
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	564 yield TokenInfo(NUMBER, token, spos, epos, line)
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	565 elif initial in '\r\n':
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	566 yield TokenInfo(NL if parenlev > 0 else NEWLINE,
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	567 token, spos, epos, line)
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	568 elif initial == '#':
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	569 assert not token.endswith("\n")
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	570 yield TokenInfo(COMMENT, token, spos, epos, line)
25016 0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	571 elif token in triple_quoted:
72853 df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	572 endprog = _compile(endpats[token])
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	573 endmatch = endprog.match(line, pos)
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	574 if endmatch: # all on one line
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	575 pos = endmatch.end(0)
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	576 token = line[start:pos]
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	577 yield TokenInfo(STRING, token, spos, (lnum, pos), line)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	578 else:
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	579 strstart = (lnum, start) # multiple lines
1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	580 contstr = line[start:]
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	581 contline = line
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	582 break
25016 0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	583 elif initial in single_quoted or \
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	584 token[:2] in single_quoted or \
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	585 token[:3] in single_quoted:
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	586 if token[-1] == '\n': # continued string
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	587 strstart = (lnum, start)
72853 df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	588 endprog = _compile(endpats[initial] or
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	589 endpats[token[1]] or
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	590 endpats[token[2]])
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	591 contstr, needcont = line[start:], 1
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	592 contline = line
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	593 break
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	594 else: # ordinary string
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	595 yield TokenInfo(STRING, token, spos, epos, line)
64318 a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	596 elif initial.isidentifier(): # ordinary name
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	597 yield TokenInfo(NAME, token, spos, epos, line)
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	598 elif initial == '\\': # continued stmt
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	599 continued = 1
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	600 else:
57210 a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	601 if initial in '([{':
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	602 parenlev += 1
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	603 elif initial in ')]}':
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	604 parenlev -= 1
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	605 yield TokenInfo(OP, token, spos, epos, line)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	606 else:
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	607 yield TokenInfo(ERRORTOKEN, line[pos],
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	608 (lnum, pos), (lnum, pos+1), line)
57210 a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	609 pos += 1
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	610
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	611 for indent in indents[1:]: # pop remaining indent levels
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	612 yield TokenInfo(DEDENT, '', (lnum, 0), (lnum, 0), '')
7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	613 yield TokenInfo(ENDMARKER, '', (lnum, 0), (lnum, 0), '')
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	614
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	615
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	616 # An undocumented, backwards compatible, API for all the places in the standard
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	617 # library that expect to be able to use tokenize with strings
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	618 def generate_tokens(readline):
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	619 return _tokenize(readline, None)
64577 2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	620
72789 85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	621 def main():
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	622 import argparse
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	623
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	624 # Helper error handling routines
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	625 def perror(message):
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	626 print(message, file=sys.stderr)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	627
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	628 def error(message, filename=None, location=None):
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	629 if location:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	630 args = (filename,) + location + (message,)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	631 perror("%s:%d:%d: error: %s" % args)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	632 elif filename:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	633 perror("%s: error: %s" % (filename, message))
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	634 else:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	635 perror("error: %s" % message)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	636 sys.exit(1)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	637
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	638 # Parse the arguments and options
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	639 parser = argparse.ArgumentParser(prog='python -m tokenize')
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	640 parser.add_argument(dest='filename', nargs='?',
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	641 metavar='filename.py',
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	642 help='the file to tokenize; defaults to stdin')
74519 f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	643 parser.add_argument('-e', '--exact', dest='exact', action='store_true',
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	644 help='display token names using the exact type')
72789 85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	645 args = parser.parse_args()
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	646
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	647 try:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	648 # Tokenize the input
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	649 if args.filename:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	650 filename = args.filename
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	651 with builtins.open(filename, 'rb') as f:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	652 tokens = list(tokenize(f.readline))
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	653 else:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	654 filename = "<stdin>"
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	655 tokens = _tokenize(sys.stdin.readline, None)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	656
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	657 # Output the tokenization
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	658 for token in tokens:
74519 f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	659 token_type = token.type
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	660 if args.exact:
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	661 token_type = token.exact_type
72789 85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	662 token_range = "%d,%d-%d,%d:" % (token.start + token.end)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	663 print("%-20s%-15s%-15r" %
74519 f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	664 (token_range, tok_name[token_type], token.string))
72789 85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	665 except IndentationError as err:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	666 line, column = err.args[1][1:3]
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	667 error(err.args[0], filename, (line, column))
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	668 except TokenError as err:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	669 line, column = err.args[1]
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	670 error(err.args[0], filename, (line, column))
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	671 except SyntaxError as err:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	672 error(err, filename)
81039 7d69d04522e3 Replace IOError with OSError (#16715) Andrew Svetlov <andrew.svetlov@gmail.com> parents: 80192 diff changeset	673 except OSError as err:
72789 85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	674 error(err)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	675 except KeyboardInterrupt:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	676 print("interrupted\n")
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	677 except Exception as err:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	678 perror("unexpected error: %s" % err)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	679 raise
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	680
64577 2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	681 if __name__ == "__main__":
72789 85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	682 main()

Mercurial > cpython-lru_cache-weakref

annotate Lib/tokenize.py @ 87690:66c1c9f32567 default tip