cpython-lru_cache-weakref: Lib/tokenize.py annotate

annotate Lib/tokenize.py @ 84007:dbfbdf2b5c19 3.3

Fix #17967: For ftp urls CWD to target instead of hopping to each directory towards target. This fixes a bug where target is accessible, but parent directories are restricted.

author	Senthil Kumaran <senthil@uthcode.com>
date	Sat, 01 Jun 2013 08:27:06 -0700
parents	1fdeddabddda
children	2dfe8262093c 7d69d04522e3

rev	line source
5173 ff4f8fa8a273 Fixed doc string, added __version__, fixed 1 bug. Guido van Rossum <guido@python.org> parents: 5172 diff changeset	1 """Tokenization help for Python programs.
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	2
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	3 tokenize(readline) is a generator that breaks a stream of bytes into
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	4 Python tokens. It decodes the bytes according to PEP-0263 for
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	5 determining source file encoding.
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	6
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	7 It accepts a readline-like method which is called repeatedly to get the
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	8 next line of input (or b"" for EOF). It generates 5-tuples with these
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	9 members:
18351 44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	10
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	11 the token type (see token.py)
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	12 the token (a string)
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	13 the starting (row, column) indices of the token (a 2-tuple of ints)
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	14 the ending (row, column) indices of the token (a 2-tuple of ints)
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	15 the original line (string)
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	16
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	17 It is designed to match the working of the Python tokenizer exactly, except
44a87c300f2b Turns out Neil didn't intend for all of his gen-branch work to get Tim Peters <tim.peters@gmail.com> parents: 18148 diff changeset	18 that it produces COMMENT tokens for comments and gives type OP for all
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	19 operators. Additionally, all token lists start with an ENCODING token
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	20 which tells you which encoding was used to decode the bytes stream.
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	21 """
5173 ff4f8fa8a273 Fixed doc string, added __version__, fixed 1 bug. Guido van Rossum <guido@python.org> parents: 5172 diff changeset	22
16899 320fa50a1a7d Add __author__ and __credits__ variables. Ka-Ping Yee <ping@zesty.ca> parents: 16889 diff changeset	23 __author__ = 'Ka-Ping Yee <ping@lfw.org>'
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	24 __credits__ = ('GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, '
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	25 'Skip Montanaro, Raymond Hettinger, Trent Nelson, '
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	26 'Michael Foord')
68096 ea260d393cde Issue #11074: Make 'tokenize' so it can be reloaded. Brett Cannon <bcannon@gmail.com> parents: 66148 diff changeset	27 import builtins
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	28 import re
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	29 import sys
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	30 from token import *
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	31 from codecs import lookup, BOM_UTF8
64581 fea86c5e3385 Experiment: Let collections.namedtuple() do the work. This should work now that _collections is pre-built. The buildbots will tell us shortly. Raymond Hettinger <python@rcn.com> parents: 64577 diff changeset	32 import collections
66097 306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	33 from io import TextIOWrapper
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	34 cookie_re = re.compile("coding[:=]\s*([-\w.]+)")
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	35
16889 0feb63490093 final round of __all__ lists (I hope) - skipped urllib2 because Moshe may be Skip Montanaro <skip@pobox.com> parents: 16514 diff changeset	36 import token
66148 98bd3ca1ed86 Issue #10386: Added __all__ to token module; this simplifies importing Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66097 diff changeset	37 __all__ = token.__all__ + ["COMMENT", "tokenize", "detect_encoding",
98bd3ca1ed86 Issue #10386: Added __all__ to token module; this simplifies importing Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 66097 diff changeset	38 "NL", "untokenize", "ENCODING", "TokenInfo"]
16889 0feb63490093 final round of __all__ lists (I hope) - skipped urllib2 because Moshe may be Skip Montanaro <skip@pobox.com> parents: 16514 diff changeset	39 del token
0feb63490093 final round of __all__ lists (I hope) - skipped urllib2 because Moshe may be Skip Montanaro <skip@pobox.com> parents: 16514 diff changeset	40
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	41 COMMENT = N_TOKENS
1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	42 tok_name[COMMENT] = 'COMMENT'
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	43 NL = N_TOKENS + 1
d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	44 tok_name[NL] = 'NL'
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	45 ENCODING = N_TOKENS + 2
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	46 tok_name[ENCODING] = 'ENCODING'
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	47 N_TOKENS += 3
74519 f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	48 EXACT_TOKEN_TYPES = {
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	49 '(': LPAR,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	50 ')': RPAR,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	51 '[': LSQB,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	52 ']': RSQB,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	53 ':': COLON,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	54 ',': COMMA,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	55 ';': SEMI,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	56 '+': PLUS,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	57 '-': MINUS,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	58 '*': STAR,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	59 '/': SLASH,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	60 '\|': VBAR,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	61 '&': AMPER,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	62 '<': LESS,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	63 '>': GREATER,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	64 '=': EQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	65 '.': DOT,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	66 '%': PERCENT,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	67 '{': LBRACE,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	68 '}': RBRACE,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	69 '==': EQEQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	70 '!=': NOTEQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	71 '<=': LESSEQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	72 '>=': GREATEREQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	73 '~': TILDE,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	74 '^': CIRCUMFLEX,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	75 '<<': LEFTSHIFT,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	76 '>>': RIGHTSHIFT,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	77 '**': DOUBLESTAR,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	78 '+=': PLUSEQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	79 '-=': MINEQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	80 '*=': STAREQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	81 '/=': SLASHEQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	82 '%=': PERCENTEQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	83 '&=': AMPEREQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	84 '\|=': VBAREQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	85 '^=': CIRCUMFLEXEQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	86 '<<=': LEFTSHIFTEQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	87 '>>=': RIGHTSHIFTEQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	88 '**=': DOUBLESTAREQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	89 '//': DOUBLESLASH,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	90 '//=': DOUBLESLASHEQUAL,
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	91 '@': AT
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	92 }
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	93
64581 fea86c5e3385 Experiment: Let collections.namedtuple() do the work. This should work now that _collections is pre-built. The buildbots will tell us shortly. Raymond Hettinger <python@rcn.com> parents: 64577 diff changeset	94 class TokenInfo(collections.namedtuple('TokenInfo', 'type string start end line')):
53597 5f81c45bdf6d Remove dependency on the collections module. Raymond Hettinger <python@rcn.com> parents: 53584 diff changeset	95 def __repr__(self):
64582 71b264e1bb71 A little bit more readable repr method. Raymond Hettinger <python@rcn.com> parents: 64581 diff changeset	96 annotated_type = '%d (%s)' % (self.type, tok_name[self.type])
71b264e1bb71 A little bit more readable repr method. Raymond Hettinger <python@rcn.com> parents: 64581 diff changeset	97 return ('TokenInfo(type=%s, string=%r, start=%r, end=%r, line=%r)' %
71b264e1bb71 A little bit more readable repr method. Raymond Hettinger <python@rcn.com> parents: 64581 diff changeset	98 self._replace(type=annotated_type))
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	99
74519 f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	100 @property
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	101 def exact_type(self):
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	102 if self.type == OP and self.string in EXACT_TOKEN_TYPES:
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	103 return EXACT_TOKEN_TYPES[self.string]
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	104 else:
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	105 return self.type
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	106
16514 90c600a819e9 String method conversion. Eric S. Raymond <esr@thyrsus.com> parents: 15898 diff changeset	107 def group(*choices): return '(' + '\|'.join(choices) + ')'
28089 14f65d661084 Get rid of many apply() calls. Guido van Rossum <guido@python.org> parents: 26032 diff changeset	108 def any(choices): return group(choices) + '*'
14f65d661084 Get rid of many apply() calls. Guido van Rossum <guido@python.org> parents: 26032 diff changeset	109 def maybe(choices): return group(choices) + '?'
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	110
48769 83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	111 # Note: we use unicode matching for names ("\w") but ascii matching for
83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	112 # number literals.
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	113 Whitespace = r'[ \f\t]*'
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	114 Comment = r'#[^\r\n]*'
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	115 Ignore = Whitespace + any(r'\\\r?\n' + Whitespace) + maybe(Comment)
64318 a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	116 Name = r'\w+'
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	117
48769 83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	118 Hexnumber = r'0[xX][0-9a-fA-F]+'
44782 ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from Georg Brandl <georg@python.org> parents: 43833 diff changeset	119 Binnumber = r'0[bB][01]+'
ca2683a67943 Merged revisions 60080-60089,60091-60093 via svnmerge from Georg Brandl <georg@python.org> parents: 43833 diff changeset	120 Octnumber = r'0[oO][0-7]+'
48769 83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	121 Decnumber = r'(?:0+\|[1-9][0-9]*)'
42084 229c28cb9afb Merged revisions 55817-55961 via svnmerge from Guido van Rossum <guido@python.org> parents: 41717 diff changeset	122 Intnumber = group(Hexnumber, Binnumber, Octnumber, Decnumber)
48769 83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	123 Exponent = r'[eE][-+]?[0-9]+'
83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	124 Pointfloat = group(r'[0-9]+\.[0-9]*', r'\.[0-9]+') + maybe(Exponent)
83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	125 Expfloat = r'[0-9]+' + Exponent
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	126 Floatnumber = group(Pointfloat, Expfloat)
48769 83b2d07b475b #2834: Change re module semantics, so that str and bytes mixing is forbidden, Antoine Pitrou <solipsis@pitrou.net> parents: 47481 diff changeset	127 Imagnumber = group(r'[0-9]+[jJ]', Floatnumber + r'[jJ]')
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	128 Number = group(Imagnumber, Floatnumber, Intnumber)
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	129
77522 8e47e9af826e Issue #15096: Drop support for the ur string prefix Christian Heimes <c.heimes@cheimes.de> parents: 77475 diff changeset	130 StringPrefix = r'(?:[bB][rR]?\|[rR][bB]?\|[uU])?'
75383 2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	131
15080 0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	132 # Tail end of ' string.
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	133 Single = r"[^'\\](?:\\.[^'\\])*'"
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	134 # Tail end of " string.
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	135 Double = r'[^"\\](?:\\.[^"\\])*"'
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	136 # Tail end of ''' string.
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	137 Single3 = r"[^'\\](?:(?:\\.\|'(?!''))[^'\\])*'''"
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	138 # Tail end of """ string.
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	139 Double3 = r'[^"\\](?:(?:\\.\|"(?!""))[^"\\])*"""'
75383 2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	140 Triple = group(StringPrefix + "'''", StringPrefix + '"""')
15080 0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	141 # Single-line ' or " string.
75383 2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	142 String = group(StringPrefix + r"'[^\n'\\](?:\\.[^\n'\\])*'",
2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	143 StringPrefix + r'"[^\n"\\](?:\\.[^\n"\\])*"')
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	144
15080 0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	145 # Because of leftmost-then-longest match semantics, be sure to put the
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	146 # longest operators first (e.g., if = came before ==, == would get
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	147 # recognized as two instances of =).
39648 efe3f2c0da5e Killed the <> operator. You must now use !=. Guido van Rossum <guido@python.org> parents: 39610 diff changeset	148 Operator = group(r"\\=?", r">>=?", r"<<=?", r"!=",
40529 54cd47788811 PEP 3107 - Function Annotations thanks to Tony Lownds Neal Norwitz <nnorwitz@gmail.com> parents: 40450 diff changeset	149 r"//=?", r"->",
15080 0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	150 r"[+\-*/%&\|^=<>]=?",
0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	151 r"~")
14166 e9c2f830f9ba Update for augmented assignment, tested & approved by Guido. Thomas Wouters <thomas@python.org> parents: 14021 diff changeset	152
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	153 Bracket = '[][(){}]'
41230 b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis. Georg Brandl <georg@python.org> parents: 40775 diff changeset	154 Special = group(r'\r?\n', r'\.\.\.', r'[:;.,@]')
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	155 Funny = group(Operator, Bracket, Special)
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	156
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	157 PlainToken = group(Number, Funny, String, Name)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	158 Token = Ignore + PlainToken
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	159
15080 0e0ea56e4be8 Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py). Tim Peters <tim.peters@gmail.com> parents: 14166 diff changeset	160 # First (or only) line of ' or " string.
75383 2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	161 ContStr = group(StringPrefix + r"'[^\n'\\](?:\\.[^\n'\\])*" +
15898 8366fcfb6090 Add tokenizer support and tests for u'', U"", uR'', Ur"", etc. Ka-Ping Yee <ping@zesty.ca> parents: 15873 diff changeset	162 group("'", r'\\\r?\n'),
75383 2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	163 StringPrefix + r'"[^\n"\\](?:\\.[^\n"\\])*' +
15898 8366fcfb6090 Add tokenizer support and tests for u'', U"", uR'', Ur"", etc. Ka-Ping Yee <ping@zesty.ca> parents: 15873 diff changeset	164 group('"', r'\\\r?\n'))
80191 3ffff1798ed5 #16152: fix tokenize to ignore whitespace at the end of the code when no newline is found. Patch by Ned Batchelder. Ezio Melotti <ezio.melotti@gmail.com> parents: 77961 diff changeset	165 PseudoExtras = group(r'\\\r?\n\|\Z', Comment, Triple)
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	166 PseudoToken = Whitespace + group(PseudoExtras, Number, Funny, ContStr, Name)
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	167
64318 a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	168 def _compile(expr):
a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	169 return re.compile(expr, re.UNICODE)
a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	170
72853 df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	171 endpats = {"'": Single, '"': Double,
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	172 "'''": Single3, '"""': Double3,
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	173 "r'''": Single3, 'r"""': Double3,
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	174 "b'''": Single3, 'b"""': Double3,
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	175 "R'''": Single3, 'R"""': Double3,
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	176 "B'''": Single3, 'B"""': Double3,
75383 2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	177 "br'''": Single3, 'br"""': Double3,
72853 df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	178 "bR'''": Single3, 'bR"""': Double3,
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	179 "Br'''": Single3, 'Br"""': Double3,
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	180 "BR'''": Single3, 'BR"""': Double3,
75383 2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	181 "rb'''": Single3, 'rb"""': Double3,
2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	182 "Rb'''": Single3, 'Rb"""': Double3,
2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	183 "rB'''": Single3, 'rB"""': Double3,
2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	184 "RB'''": Single3, 'RB"""': Double3,
75381 ed9497d6a216 Basic support for PEP 414 without docs or tests. Armin Ronacher <armin.ronacher@active-4.com> parents: 74519 diff changeset	185 "u'''": Single3, 'u"""': Double3,
ed9497d6a216 Basic support for PEP 414 without docs or tests. Armin Ronacher <armin.ronacher@active-4.com> parents: 74519 diff changeset	186 "R'''": Single3, 'R"""': Double3,
ed9497d6a216 Basic support for PEP 414 without docs or tests. Armin Ronacher <armin.ronacher@active-4.com> parents: 74519 diff changeset	187 "U'''": Single3, 'U"""': Double3,
ed9497d6a216 Basic support for PEP 414 without docs or tests. Armin Ronacher <armin.ronacher@active-4.com> parents: 74519 diff changeset	188 'r': None, 'R': None, 'b': None, 'B': None,
ed9497d6a216 Basic support for PEP 414 without docs or tests. Armin Ronacher <armin.ronacher@active-4.com> parents: 74519 diff changeset	189 'u': None, 'U': None}
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	190
25016 0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	191 triple_quoted = {}
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	192 for t in ("'''", '"""',
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	193 "r'''", 'r"""', "R'''", 'R"""',
43833 ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	194 "b'''", 'b"""', "B'''", 'B"""',
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	195 "br'''", 'br"""', "Br'''", 'Br"""',
75381 ed9497d6a216 Basic support for PEP 414 without docs or tests. Armin Ronacher <armin.ronacher@active-4.com> parents: 74519 diff changeset	196 "bR'''", 'bR"""', "BR'''", 'BR"""',
75383 2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	197 "rb'''", 'rb"""', "rB'''", 'rB"""',
2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	198 "Rb'''", 'Rb"""', "RB'''", 'RB"""',
75381 ed9497d6a216 Basic support for PEP 414 without docs or tests. Armin Ronacher <armin.ronacher@active-4.com> parents: 74519 diff changeset	199 "u'''", 'u"""', "U'''", 'U"""',
77522 8e47e9af826e Issue #15096: Drop support for the ur string prefix Christian Heimes <c.heimes@cheimes.de> parents: 77475 diff changeset	200 ):
25016 0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	201 triple_quoted[t] = t
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	202 single_quoted = {}
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	203 for t in ("'", '"',
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	204 "r'", 'r"', "R'", 'R"',
43833 ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	205 "b'", 'b"', "B'", 'B"',
ce77d91ea1d0 Patch 1420 by Ron Adam. Guido van Rossum <guido@python.org> parents: 43020 diff changeset	206 "br'", 'br"', "Br'", 'Br"',
75381 ed9497d6a216 Basic support for PEP 414 without docs or tests. Armin Ronacher <armin.ronacher@active-4.com> parents: 74519 diff changeset	207 "bR'", 'bR"', "BR'", 'BR"' ,
75383 2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	208 "rb'", 'rb"', "rB'", 'rB"',
2822765e48a7 Updated tokenize to support the inverse byte literals new in 3.3 Armin Ronacher <armin.ronacher@active-4.com> parents: 75381 diff changeset	209 "Rb'", 'Rb"', "RB'", 'RB"' ,
75381 ed9497d6a216 Basic support for PEP 414 without docs or tests. Armin Ronacher <armin.ronacher@active-4.com> parents: 74519 diff changeset	210 "u'", 'u"', "U'", 'U"',
77522 8e47e9af826e Issue #15096: Drop support for the ur string prefix Christian Heimes <c.heimes@cheimes.de> parents: 77475 diff changeset	211 ):
25016 0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	212 single_quoted[t] = t
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	213
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	214 tabsize = 8
14021 5b34b70ecf22 Convert some old-style string exceptions to class exceptions. Fred Drake <fdrake@acm.org> parents: 7839 diff changeset	215
17178 707f95d05e69 Provide a StopTokenizing exception for conveniently exiting the loop. Ka-Ping Yee <ping@zesty.ca> parents: 16901 diff changeset	216 class TokenError(Exception): pass
707f95d05e69 Provide a StopTokenizing exception for conveniently exiting the loop. Ka-Ping Yee <ping@zesty.ca> parents: 16901 diff changeset	217
707f95d05e69 Provide a StopTokenizing exception for conveniently exiting the loop. Ka-Ping Yee <ping@zesty.ca> parents: 16901 diff changeset	218 class StopTokenizing(Exception): pass
14021 5b34b70ecf22 Convert some old-style string exceptions to class exceptions. Fred Drake <fdrake@acm.org> parents: 7839 diff changeset	219
18148 7bc473b2c654 Merging the gen-branch into the main line, at Guido's direction. Yay! Tim Peters <tim.peters@gmail.com> parents: 17178 diff changeset	220
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	221 class Untokenizer:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	222
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	223 def __init__(self):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	224 self.tokens = []
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	225 self.prev_row = 1
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	226 self.prev_col = 0
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	227 self.encoding = None
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	228
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	229 def add_whitespace(self, start):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	230 row, col = start
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	231 assert row <= self.prev_row
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	232 col_offset = col - self.prev_col
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	233 if col_offset:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	234 self.tokens.append(" " * col_offset)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	235
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	236 def untokenize(self, iterable):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	237 for t in iterable:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	238 if len(t) == 2:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	239 self.compat(t, iterable)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	240 break
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	241 tok_type, token, start, end, line = t
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	242 if tok_type == ENCODING:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	243 self.encoding = token
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	244 continue
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	245 self.add_whitespace(start)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	246 self.tokens.append(token)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	247 self.prev_row, self.prev_col = end
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	248 if tok_type in (NEWLINE, NL):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	249 self.prev_row += 1
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	250 self.prev_col = 0
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	251 return "".join(self.tokens)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	252
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	253 def compat(self, token, iterable):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	254 startline = False
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	255 indents = []
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	256 toks_append = self.tokens.append
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	257 toknum, tokval = token
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	258
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	259 if toknum in (NAME, NUMBER):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	260 tokval += ' '
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	261 if toknum in (NEWLINE, NL):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	262 startline = True
46145 da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	263 prevstring = False
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	264 for tok in iterable:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	265 toknum, tokval = tok[:2]
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	266 if toknum == ENCODING:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	267 self.encoding = tokval
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	268 continue
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	269
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	270 if toknum in (NAME, NUMBER):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	271 tokval += ' '
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	272
46145 da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	273 # Insert a space between two consecutive strings
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	274 if toknum == STRING:
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	275 if prevstring:
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	276 tokval = ' ' + tokval
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	277 prevstring = True
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	278 else:
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	279 prevstring = False
da5597a5c195 Merged revisions 61964-61979 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 45888 diff changeset	280
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	281 if toknum == INDENT:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	282 indents.append(tokval)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	283 continue
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	284 elif toknum == DEDENT:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	285 indents.pop()
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	286 continue
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	287 elif toknum in (NEWLINE, NL):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	288 startline = True
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	289 elif startline and indents:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	290 toks_append(indents[-1])
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	291 startline = False
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	292 toks_append(tokval)
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	293
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	294
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	295 def untokenize(iterable):
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	296 """Transform tokens back into Python source code.
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	297 It returns a bytes object, encoded using the ENCODING
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	298 token, which is the first token sequence output by tokenize.
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	299
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	300 Each element returned by the iterable must be a token sequence
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	301 with at least two elements, a token number and token value. If
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	302 only two tokens are passed, the resulting output is poor.
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	303
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	304 Round-trip invariant for full input:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	305 Untokenized source will match input source exactly
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	306
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	307 Round-trip invariant for limited intput:
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	308 # Output bytes will tokenize the back to the input
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	309 t1 = [tok[:2] for tok in tokenize(f.readline)]
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	310 newcode = untokenize(t1)
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	311 readline = BytesIO(newcode).readline
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	312 t2 = [tok[:2] for tok in tokenize(readline)]
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	313 assert t1 == t2
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	314 """
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	315 ut = Untokenizer()
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	316 out = ut.untokenize(iterable)
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	317 if ut.encoding is not None:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	318 out = out.encode(ut.encoding)
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	319 return out
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	320
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	321
56434 12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	322 def _get_normal_name(orig_enc):
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	323 """Imitates get_normal_name in tokenizer.c."""
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	324 # Only care about the first 12 characters.
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	325 enc = orig_enc[:12].lower().replace("_", "-")
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	326 if enc == "utf-8" or enc.startswith("utf-8-"):
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	327 return "utf-8"
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	328 if enc in ("latin-1", "iso-8859-1", "iso-latin-1") or \
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	329 enc.startswith(("latin-1-", "iso-8859-1-", "iso-latin-1-")):
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	330 return "iso-8859-1"
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	331 return orig_enc
12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	332
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	333 def detect_encoding(readline):
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	334 """
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	335 The detect_encoding() function is used to detect the encoding that should
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	336 be used to decode a Python source file. It requires one argment, readline,
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	337 in the same way as the tokenize() generator.
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	338
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	339 It will call readline a maximum of twice, and return the encoding used
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	340 (as a string) and a list of any lines (left as bytes) it has read in.
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	341
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	342 It detects the encoding from the presence of a utf-8 bom or an encoding
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	343 cookie as specified in pep-0263. If both a bom and a cookie are present,
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	344 but disagree, a SyntaxError will be raised. If the encoding cookie is an
1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	345 invalid charset, raise a SyntaxError. Note that if a utf-8 bom is found,
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	346 'utf-8-sig' is returned.
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	347
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	348 If no encoding is specified, then the default of 'utf-8' will be returned.
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	349 """
76430 1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	350 try:
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	351 filename = readline.__self__.name
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	352 except AttributeError:
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	353 filename = None
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	354 bom_found = False
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	355 encoding = None
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	356 default = 'utf-8'
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	357 def read_or_stop():
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	358 try:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	359 return readline()
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	360 except StopIteration:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	361 return b''
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	362
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	363 def find_cookie(line):
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	364 try:
76425 b07488490001 Issue #14629: Raise SyntaxError in tokenizer.detect_encoding Martin v. Löwis <martin@v.loewis.de> parents: 68257 diff changeset	365 # Decode as UTF-8. Either the line is an encoding declaration,
b07488490001 Issue #14629: Raise SyntaxError in tokenizer.detect_encoding Martin v. Löwis <martin@v.loewis.de> parents: 68257 diff changeset	366 # in which case it should be pure ASCII, or it must be UTF-8
b07488490001 Issue #14629: Raise SyntaxError in tokenizer.detect_encoding Martin v. Löwis <martin@v.loewis.de> parents: 68257 diff changeset	367 # per default encoding.
b07488490001 Issue #14629: Raise SyntaxError in tokenizer.detect_encoding Martin v. Löwis <martin@v.loewis.de> parents: 68257 diff changeset	368 line_string = line.decode('utf-8')
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	369 except UnicodeDecodeError:
76430 1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	370 msg = "invalid or missing encoding declaration"
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	371 if filename is not None:
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	372 msg = '{} for {!r}'.format(msg, filename)
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	373 raise SyntaxError(msg)
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	374
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	375 matches = cookie_re.findall(line_string)
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	376 if not matches:
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	377 return None
56434 12cd94d39874 Merged revisions 75299 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 53597 diff changeset	378 encoding = _get_normal_name(matches[0])
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	379 try:
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	380 codec = lookup(encoding)
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	381 except LookupError:
df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	382 # This behaviour mimics the Python interpreter
76430 1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	383 if filename is None:
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	384 msg = "unknown encoding: " + encoding
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	385 else:
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	386 msg = "unknown encoding for {!r}: {}".format(filename,
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	387 encoding)
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	388 raise SyntaxError(msg)
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	389
59701 15b87ea1413d fix for files with coding cookies and BOMs Benjamin Peterson <benjamin@python.org> parents: 59700 diff changeset	390 if bom_found:
77961 5020afc0b7c9 Issue #14990: tokenize: correctly fail with SyntaxError on invalid encoding declaration. Florent Xicluna <florent.xicluna@gmail.com> parents: 76425 diff changeset	391 if encoding != 'utf-8':
59701 15b87ea1413d fix for files with coding cookies and BOMs Benjamin Peterson <benjamin@python.org> parents: 59700 diff changeset	392 # This behaviour mimics the Python interpreter
76430 1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	393 if filename is None:
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	394 msg = 'encoding problem: utf-8'
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	395 else:
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	396 msg = 'encoding problem for {!r}: utf-8'.format(filename)
1b57de8a8383 Issue #14629: Mention the filename in SyntaxError exceptions from Brett Cannon <brett@python.org> parents: 76426 diff changeset	397 raise SyntaxError(msg)
59701 15b87ea1413d fix for files with coding cookies and BOMs Benjamin Peterson <benjamin@python.org> parents: 59700 diff changeset	398 encoding += '-sig'
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	399 return encoding
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	400
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	401 first = read_or_stop()
50127 df6649951410 Merged revisions 67711 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 48769 diff changeset	402 if first.startswith(BOM_UTF8):
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	403 bom_found = True
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	404 first = first[3:]
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	405 default = 'utf-8-sig'
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	406 if not first:
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	407 return default, []
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	408
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	409 encoding = find_cookie(first)
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	410 if encoding:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	411 return encoding, [first]
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	412
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	413 second = read_or_stop()
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	414 if not second:
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	415 return default, [first]
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	416
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	417 encoding = find_cookie(second)
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	418 if encoding:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	419 return encoding, [first, second]
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	420
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	421 return default, [first, second]
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	422
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	423
66097 306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	424 def open(filename):
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	425 """Open a file in read only mode using the encoding detected by
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	426 detect_encoding().
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	427 """
68096 ea260d393cde Issue #11074: Make 'tokenize' so it can be reloaded. Brett Cannon <bcannon@gmail.com> parents: 66148 diff changeset	428 buffer = builtins.open(filename, 'rb')
66097 306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	429 encoding, lines = detect_encoding(buffer.readline)
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	430 buffer.seek(0)
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	431 text = TextIOWrapper(buffer, encoding, line_buffering=True)
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	432 text.mode = 'r'
306339c31758 Issue #10335: Add tokenize.open(), detect the file encoding using Victor Stinner <victor.stinner@haypocalc.com> parents: 64582 diff changeset	433 return text
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	434
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	435
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	436 def tokenize(readline):
23455 68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	437 """
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	438 The tokenize() generator requires one argment, readline, which
23455 68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	439 must be a callable object which provides the same interface as the
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	440 readline() method of built-in file objects. Each call to the function
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	441 should return one line of input as bytes. Alternately, readline
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	442 can be a callable function terminating with StopIteration:
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	443 readline = open(myfile, 'rb').__next__ # Example of alternate readline
23521 a90faa5f9417 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 23455 diff changeset	444
23455 68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	445 The generator produces 5-tuples with these members: the token type; the
68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	446 token string; a 2-tuple (srow, scol) of ints specifying the row and
68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	447 column where the token begins in the source; a 2-tuple (erow, ecol) of
68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	448 ints specifying the row and column where the token ends in the source;
64406 1f47206d577f Remove unused import, fix typo and rewrap docstrings. Florent Xicluna <florent.xicluna@gmail.com> parents: 64317 diff changeset	449 and the line on which the token was found. The line passed is the
23521 a90faa5f9417 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 23455 diff changeset	450 logical line; continuation lines are included.
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	451
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	452 The first token sequence will always be an ENCODING token
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	453 which tells you which encoding was used to decode the bytes stream.
23455 68c776ba5ea5 Added docstrings excerpted from Python Library Reference. Raymond Hettinger <python@rcn.com> parents: 22719 diff changeset	454 """
57235 555045541794 simply by using itertools.chain() Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	455 # This import is here to avoid problems when the itertools module is not
555045541794 simply by using itertools.chain() Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	456 # built yet and tokenize is imported.
57237 94f2705def5c use some more itertools magic to make '' be yielded after readline is done Benjamin Peterson <benjamin@python.org> parents: 57235 diff changeset	457 from itertools import chain, repeat
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	458 encoding, consumed = detect_encoding(readline)
57237 94f2705def5c use some more itertools magic to make '' be yielded after readline is done Benjamin Peterson <benjamin@python.org> parents: 57235 diff changeset	459 rl_gen = iter(readline, b"")
94f2705def5c use some more itertools magic to make '' be yielded after readline is done Benjamin Peterson <benjamin@python.org> parents: 57235 diff changeset	460 empty = repeat(b"")
94f2705def5c use some more itertools magic to make '' be yielded after readline is done Benjamin Peterson <benjamin@python.org> parents: 57235 diff changeset	461 return _tokenize(chain(consumed, rl_gen, empty).__next__, encoding)
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	462
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	463
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	464 def _tokenize(readline, encoding):
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	465 lnum = parenlev = continued = 0
64318 a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	466 numchars = '0123456789'
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	467 contstr, needcont = '', 0
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	468 contline = None
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	469 indents = [0]
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	470
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	471 if encoding is not None:
59700 72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	472 if encoding == "utf-8-sig":
72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	473 # BOM will already have been stripped.
72a809d165f4 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found Benjamin Peterson <benjamin@python.org> parents: 57237 diff changeset	474 encoding = "utf-8"
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	475 yield TokenInfo(ENCODING, encoding, (0, 0), (0, 0), '')
47481 0d7ccb50007c use the more idomatic (and Py3k faster) while True Benjamin Peterson <benjamin@python.org> parents: 46145 diff changeset	476 while True: # loop over lines in stream
35058 8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	477 try:
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	478 line = readline()
8076424d455d Add untokenize() function to allow full round-trip tokenization. Raymond Hettinger <python@rcn.com> parents: 32888 diff changeset	479 except StopIteration:
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	480 line = b''
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	481
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	482 if encoding is not None:
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	483 line = line.decode(encoding)
57210 a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	484 lnum += 1
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	485 pos, max = 0, len(line)
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	486
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	487 if contstr: # continued string
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	488 if not line:
43020 907a45df1295 Raise statement normalization in Lib/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	489 raise TokenError("EOF in multi-line string", strstart)
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	490 endmatch = endprog.match(line)
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	491 if endmatch:
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	492 pos = end = endmatch.end(0)
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	493 yield TokenInfo(STRING, contstr + line[:end],
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	494 strstart, (lnum, end), contline + line)
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	495 contstr, needcont = '', 0
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	496 contline = None
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	497 elif needcont and line[-2:] != '\\\n' and line[-3:] != '\\\r\n':
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	498 yield TokenInfo(ERRORTOKEN, contstr + line,
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	499 strstart, (lnum, len(line)), contline)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	500 contstr = ''
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	501 contline = None
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	502 continue
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	503 else:
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	504 contstr = contstr + line
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	505 contline = contline + line
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	506 continue
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	507
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	508 elif parenlev == 0 and not continued: # new statement
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	509 if not line: break
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	510 column = 0
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	511 while pos < max: # measure leading whitespace
57210 a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	512 if line[pos] == ' ':
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	513 column += 1
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	514 elif line[pos] == '\t':
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	515 column = (column//tabsize + 1)*tabsize
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	516 elif line[pos] == '\f':
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	517 column = 0
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	518 else:
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	519 break
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	520 pos += 1
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	521 if pos == max:
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	522 break
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	523
1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	524 if line[pos] in '#\r\n': # skip comments or blank lines
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	525 if line[pos] == '#':
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	526 comment_token = line[pos:].rstrip('\r\n')
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	527 nl_pos = pos + len(comment_token)
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	528 yield TokenInfo(COMMENT, comment_token,
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	529 (lnum, pos), (lnum, pos + len(comment_token)), line)
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	530 yield TokenInfo(NL, line[nl_pos:],
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	531 (lnum, nl_pos), (lnum, len(line)), line)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	532 else:
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	533 yield TokenInfo((NL, COMMENT)[line[pos] == '#'], line[pos:],
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	534 (lnum, pos), (lnum, len(line)), line)
1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	535 continue
558 cfb53322c9d1 Initial revision Guido van Rossum <guido@python.org> parents: diff changeset	536
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	537 if column > indents[-1]: # count indents or dedents
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	538 indents.append(column)
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	539 yield TokenInfo(INDENT, line[:pos], (lnum, 0), (lnum, pos), line)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	540 while column < indents[-1]:
35119 04868959f1fe SF bug #1224621: tokenize module does not detect inconsistent dedents Raymond Hettinger <python@rcn.com> parents: 35058 diff changeset	541 if column not in indents:
04868959f1fe SF bug #1224621: tokenize module does not detect inconsistent dedents Raymond Hettinger <python@rcn.com> parents: 35058 diff changeset	542 raise IndentationError(
39610 3674f20ac781 Merge current trunk into p3yk. This includes the PyNumber_Index API change, Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	543 "unindent does not match any outer indentation level",
3674f20ac781 Merge current trunk into p3yk. This includes the PyNumber_Index API change, Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	544 ("<tokenize>", lnum, pos, line))
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	545 indents = indents[:-1]
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	546 yield TokenInfo(DEDENT, '', (lnum, pos), (lnum, pos), line)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	547
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	548 else: # continued statement
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	549 if not line:
43020 907a45df1295 Raise statement normalization in Lib/. Collin Winter <collinw@gmail.com> parents: 42084 diff changeset	550 raise TokenError("EOF in multi-line statement", (lnum, 0))
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	551 continued = 0
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	552
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	553 while pos < max:
72853 df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	554 pseudomatch = _compile(PseudoToken).match(line, pos)
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	555 if pseudomatch: # scan for tokens
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	556 start, end = pseudomatch.span(1)
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	557 spos, epos, pos = (lnum, start), (lnum, end), end
80191 3ffff1798ed5 #16152: fix tokenize to ignore whitespace at the end of the code when no newline is found. Patch by Ned Batchelder. Ezio Melotti <ezio.melotti@gmail.com> parents: 77961 diff changeset	558 if start == end:
3ffff1798ed5 #16152: fix tokenize to ignore whitespace at the end of the code when no newline is found. Patch by Ned Batchelder. Ezio Melotti <ezio.melotti@gmail.com> parents: 77961 diff changeset	559 continue
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	560 token, initial = line[start:end], line[start]
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	561
41230 b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis. Georg Brandl <georg@python.org> parents: 40775 diff changeset	562 if (initial in numchars or # ordinary number
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis. Georg Brandl <georg@python.org> parents: 40775 diff changeset	563 (initial == '.' and token != '.' and token != '...')):
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	564 yield TokenInfo(NUMBER, token, spos, epos, line)
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	565 elif initial in '\r\n':
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	566 yield TokenInfo(NL if parenlev > 0 else NEWLINE,
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	567 token, spos, epos, line)
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	568 elif initial == '#':
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39676 diff changeset	569 assert not token.endswith("\n")
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	570 yield TokenInfo(COMMENT, token, spos, epos, line)
25016 0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	571 elif token in triple_quoted:
72853 df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	572 endprog = _compile(endpats[token])
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	573 endmatch = endprog.match(line, pos)
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	574 if endmatch: # all on one line
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	575 pos = endmatch.end(0)
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	576 token = line[start:pos]
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	577 yield TokenInfo(STRING, token, spos, (lnum, pos), line)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	578 else:
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	579 strstart = (lnum, start) # multiple lines
1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	580 contstr = line[start:]
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	581 contline = line
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	582 break
25016 0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	583 elif initial in single_quoted or \
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	584 token[:2] in single_quoted or \
0ebec83eb3dd Speed up the most egregious "if token in (long tuple)" cases by using Guido van Rossum <guido@python.org> parents: 23521 diff changeset	585 token[:3] in single_quoted:
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	586 if token[-1] == '\n': # continued string
5266 1efc4273fdb7 Ka-Ping's muich improved version of March 26, 1997: Guido van Rossum <guido@python.org> parents: 5178 diff changeset	587 strstart = (lnum, start)
72853 df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	588 endprog = _compile(endpats[initial] or
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	589 endpats[token[1]] or
df950158dc33 Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore. Antoine Pitrou <solipsis@pitrou.net> parents: 72789 diff changeset	590 endpats[token[2]])
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	591 contstr, needcont = line[start:], 1
7839 d4f5321b9d9e Differentiate between NEWLINE token (an official newline) and NL token Guido van Rossum <guido@python.org> parents: 6532 diff changeset	592 contline = line
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	593 break
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	594 else: # ordinary string
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	595 yield TokenInfo(STRING, token, spos, epos, line)
64318 a492f1ac0575 Merged revisions 84364 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 57210 diff changeset	596 elif initial.isidentifier(): # ordinary name
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	597 yield TokenInfo(NAME, token, spos, epos, line)
6531 e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	598 elif initial == '\\': # continued stmt
e497a7fc0aa8 Redone (by Ka-Ping) using the new re module, and adding recognition Guido van Rossum <guido@python.org> parents: 5721 diff changeset	599 continued = 1
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	600 else:
57210 a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	601 if initial in '([{':
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	602 parenlev += 1
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	603 elif initial in ')]}':
a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	604 parenlev -= 1
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	605 yield TokenInfo(OP, token, spos, epos, line)
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	606 else:
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	607 yield TokenInfo(ERRORTOKEN, line[pos],
5277 7df5a0d1f523 Ping's latest. Fixes triple quoted strings ending in odd Guido van Rossum <guido@python.org> parents: 5266 diff changeset	608 (lnum, pos), (lnum, pos+1), line)
57210 a0951f11be3d Merged revisions 76235 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 56434 diff changeset	609 pos += 1
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	610
31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	611 for indent in indents[1:]: # pop remaining indent levels
53584 7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	612 yield TokenInfo(DEDENT, '', (lnum, 0), (lnum, 0), '')
7fda5af2af7d Issue #5857: tokenize.tokenize() now returns named tuples. Raymond Hettinger <python@rcn.com> parents: 52325 diff changeset	613 yield TokenInfo(ENDMARKER, '', (lnum, 0), (lnum, 0), '')
5172 31b0e5f0badf Ka-Ping's version. Guido van Rossum <guido@python.org> parents: 1407 diff changeset	614
45888 51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	615
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	616 # An undocumented, backwards compatible, API for all the places in the standard
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	617 # library that expect to be able to use tokenize with strings
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	618 def generate_tokens(readline):
51e24512e305 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been Trent Nelson <trent.nelson@snakebite.org> parents: 44782 diff changeset	619 return _tokenize(readline, None)
64577 2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	620
72789 85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	621 def main():
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	622 import argparse
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	623
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	624 # Helper error handling routines
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	625 def perror(message):
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	626 print(message, file=sys.stderr)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	627
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	628 def error(message, filename=None, location=None):
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	629 if location:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	630 args = (filename,) + location + (message,)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	631 perror("%s:%d:%d: error: %s" % args)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	632 elif filename:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	633 perror("%s: error: %s" % (filename, message))
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	634 else:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	635 perror("error: %s" % message)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	636 sys.exit(1)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	637
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	638 # Parse the arguments and options
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	639 parser = argparse.ArgumentParser(prog='python -m tokenize')
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	640 parser.add_argument(dest='filename', nargs='?',
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	641 metavar='filename.py',
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	642 help='the file to tokenize; defaults to stdin')
74519 f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	643 parser.add_argument('-e', '--exact', dest='exact', action='store_true',
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	644 help='display token names using the exact type')
72789 85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	645 args = parser.parse_args()
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	646
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	647 try:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	648 # Tokenize the input
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	649 if args.filename:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	650 filename = args.filename
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	651 with builtins.open(filename, 'rb') as f:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	652 tokens = list(tokenize(f.readline))
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	653 else:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	654 filename = "<stdin>"
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	655 tokens = _tokenize(sys.stdin.readline, None)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	656
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	657 # Output the tokenization
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	658 for token in tokens:
74519 f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	659 token_type = token.type
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	660 if args.exact:
f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	661 token_type = token.exact_type
72789 85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	662 token_range = "%d,%d-%d,%d:" % (token.start + token.end)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	663 print("%-20s%-15s%-15r" %
74519 f4976fa6e830 Issue #2134: Add support for tokenize.TokenInfo.exact_type. Meador Inge <meadori@gmail.com> parents: 72853 diff changeset	664 (token_range, tok_name[token_type], token.string))
72789 85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	665 except IndentationError as err:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	666 line, column = err.args[1][1:3]
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	667 error(err.args[0], filename, (line, column))
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	668 except TokenError as err:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	669 line, column = err.args[1]
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	670 error(err.args[0], filename, (line, column))
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	671 except SyntaxError as err:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	672 error(err, filename)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	673 except IOError as err:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	674 error(err)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	675 except KeyboardInterrupt:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	676 print("interrupted\n")
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	677 except Exception as err:
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	678 perror("unexpected error: %s" % err)
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	679 raise
85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	680
64577 2c547210f8c9 Improve the repr for the TokenInfo named tuple. Raymond Hettinger <python@rcn.com> parents: 64406 diff changeset	681 if __name__ == "__main__":
72789 85254bb6c1c6 Issue #12943: python -m tokenize support has been added to tokenize. Meador Inge <meadori@gmail.com> parents: 68257 diff changeset	682 main()

Mercurial > cpython-lru_cache-weakref

annotate Lib/tokenize.py @ 84007:dbfbdf2b5c19 3.3