cpython-lru_cache-weakref: Lib/token.py annotate

annotate Lib/token.py @ 70817:dd775b1561ba

Fix markup.

author	Georg Brandl <georg@python.org>
date	Tue, 14 Jun 2011 21:09:55 +0200
parents	98bd3ca1ed86
children	0f7253d6b1cc

rev	line source
66148 98bd3ca1ed86 Issue #10386: Added __all__ to token module; this simplifies importing Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 59522 diff changeset	1 """Token constants (from "token.h")."""
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 7755 diff changeset	2
66148 98bd3ca1ed86 Issue #10386: Added __all__ to token module; this simplifies importing Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 59522 diff changeset	3 __all__ = ['tok_name', 'ISTERMINAL', 'ISNONTERMINAL', 'ISEOF']
11707 6f4146846337 The third and final doc-string sweep by Ka-Ping Yee. Guido van Rossum <guido@python.org> parents: 7755 diff changeset	4
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	5 # This file is automatically generated; please don't muck it up!
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	6 #
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	7 # To update the symbols in this file, 'cd' to the top directory of
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	8 # the python source tree after building the interpreter and run:
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	9 #
6519 24da035ceb6e Convert all remaining simple cases of regex usage to re usage. Guido van Rossum <guido@python.org> parents: 6406 diff changeset	10 # python Lib/token.py
1407 b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	11
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	12 #--start constants--
1407 b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	13 ENDMARKER = 0
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	14 NAME = 1
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	15 NUMBER = 2
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	16 STRING = 3
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	17 NEWLINE = 4
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	18 INDENT = 5
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	19 DEDENT = 6
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	20 LPAR = 7
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	21 RPAR = 8
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	22 LSQB = 9
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	23 RSQB = 10
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	24 COLON = 11
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	25 COMMA = 12
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	26 SEMI = 13
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	27 PLUS = 14
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	28 MINUS = 15
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	29 STAR = 16
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	30 SLASH = 17
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	31 VBAR = 18
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	32 AMPER = 19
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	33 LESS = 20
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	34 GREATER = 21
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	35 EQUAL = 22
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	36 DOT = 23
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	37 PERCENT = 24
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	38 LBRACE = 26
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	39 RBRACE = 27
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	40 EQEQUAL = 28
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	41 NOTEQUAL = 29
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	42 LESSEQUAL = 30
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	43 GREATEREQUAL = 31
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	44 TILDE = 32
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	45 CIRCUMFLEX = 33
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	46 LEFTSHIFT = 34
b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	47 RIGHTSHIFT = 35
3754 d23fea248a1e Changes for new parser module (Fred Drake) Guido van Rossum <guido@python.org> parents: 1407 diff changeset	48 DOUBLESTAR = 36
14165 01846568d147 Update for augmented assignment. Thomas Wouters <thomas@python.org> parents: 11707 diff changeset	49 PLUSEQUAL = 37
01846568d147 Update for augmented assignment. Thomas Wouters <thomas@python.org> parents: 11707 diff changeset	50 MINEQUAL = 38
01846568d147 Update for augmented assignment. Thomas Wouters <thomas@python.org> parents: 11707 diff changeset	51 STAREQUAL = 39
01846568d147 Update for augmented assignment. Thomas Wouters <thomas@python.org> parents: 11707 diff changeset	52 SLASHEQUAL = 40
01846568d147 Update for augmented assignment. Thomas Wouters <thomas@python.org> parents: 11707 diff changeset	53 PERCENTEQUAL = 41
01846568d147 Update for augmented assignment. Thomas Wouters <thomas@python.org> parents: 11707 diff changeset	54 AMPEREQUAL = 42
01846568d147 Update for augmented assignment. Thomas Wouters <thomas@python.org> parents: 11707 diff changeset	55 VBAREQUAL = 43
01846568d147 Update for augmented assignment. Thomas Wouters <thomas@python.org> parents: 11707 diff changeset	56 CIRCUMFLEXEQUAL = 44
01846568d147 Update for augmented assignment. Thomas Wouters <thomas@python.org> parents: 11707 diff changeset	57 LEFTSHIFTEQUAL = 45
01846568d147 Update for augmented assignment. Thomas Wouters <thomas@python.org> parents: 11707 diff changeset	58 RIGHTSHIFTEQUAL = 46
01846568d147 Update for augmented assignment. Thomas Wouters <thomas@python.org> parents: 11707 diff changeset	59 DOUBLESTAREQUAL = 47
19059 c979625feb49 Regenerated token.py to account for new DOUBLESLASH and DOUBLESLASHEQUAL. Tim Peters <tim.peters@gmail.com> parents: 16548 diff changeset	60 DOUBLESLASH = 48
c979625feb49 Regenerated token.py to account for new DOUBLESLASH and DOUBLESLASHEQUAL. Tim Peters <tim.peters@gmail.com> parents: 16548 diff changeset	61 DOUBLESLASHEQUAL = 49
32888 c7e03cf8333e PEP-0318, @decorator-style. In Guido's words: Anthony Baxter <anthonybaxter@gmail.com> parents: 25537 diff changeset	62 AT = 50
40529 54cd47788811 PEP 3107 - Function Annotations thanks to Tony Lownds Neal Norwitz <nnorwitz@gmail.com> parents: 39698 diff changeset	63 RARROW = 51
41230 b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis. Georg Brandl <georg@python.org> parents: 40928 diff changeset	64 ELLIPSIS = 52
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis. Georg Brandl <georg@python.org> parents: 40928 diff changeset	65 OP = 53
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis. Georg Brandl <georg@python.org> parents: 40928 diff changeset	66 ERRORTOKEN = 54
b1a18c7ed5ca Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis. Georg Brandl <georg@python.org> parents: 40928 diff changeset	67 N_TOKENS = 55
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	68 NT_OFFSET = 256
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	69 #--end constants--
1407 b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	70
66148 98bd3ca1ed86 Issue #10386: Added __all__ to token module; this simplifies importing Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 59522 diff changeset	71 tok_name = {value: name
98bd3ca1ed86 Issue #10386: Added __all__ to token module; this simplifies importing Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 59522 diff changeset	72 for name, value in globals().items()
98bd3ca1ed86 Issue #10386: Added __all__ to token module; this simplifies importing Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 59522 diff changeset	73 if isinstance(value, int)}
98bd3ca1ed86 Issue #10386: Added __all__ to token module; this simplifies importing Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 59522 diff changeset	74 __all__.extend(tok_name.values())
1407 b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	75
3754 d23fea248a1e Changes for new parser module (Fred Drake) Guido van Rossum <guido@python.org> parents: 1407 diff changeset	76 def ISTERMINAL(x):
d23fea248a1e Changes for new parser module (Fred Drake) Guido van Rossum <guido@python.org> parents: 1407 diff changeset	77 return x < NT_OFFSET
1407 b9fcfb99a3e5 * test_select.py: (some) tests for built-in select module Guido van Rossum <guido@python.org> parents: diff changeset	78
3754 d23fea248a1e Changes for new parser module (Fred Drake) Guido van Rossum <guido@python.org> parents: 1407 diff changeset	79 def ISNONTERMINAL(x):
d23fea248a1e Changes for new parser module (Fred Drake) Guido van Rossum <guido@python.org> parents: 1407 diff changeset	80 return x >= NT_OFFSET
d23fea248a1e Changes for new parser module (Fred Drake) Guido van Rossum <guido@python.org> parents: 1407 diff changeset	81
d23fea248a1e Changes for new parser module (Fred Drake) Guido van Rossum <guido@python.org> parents: 1407 diff changeset	82 def ISEOF(x):
d23fea248a1e Changes for new parser module (Fred Drake) Guido van Rossum <guido@python.org> parents: 1407 diff changeset	83 return x == ENDMARKER
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	84
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	85
66148 98bd3ca1ed86 Issue #10386: Added __all__ to token module; this simplifies importing Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 59522 diff changeset	86 def _main():
6519 24da035ceb6e Convert all remaining simple cases of regex usage to re usage. Guido van Rossum <guido@python.org> parents: 6406 diff changeset	87 import re
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	88 import sys
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	89 args = sys.argv[1:]
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	90 inFileName = args and args[0] or "Include/token.h"
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	91 outFileName = "Lib/token.py"
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	92 if len(args) > 1:
7755 fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6519 diff changeset	93 outFileName = args[1]
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	94 try:
7755 fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6519 diff changeset	95 fp = open(inFileName)
40587 6b18a095c2e7 SF patch 1631942 by Collin Winter: Guido van Rossum <guido@python.org> parents: 40529 diff changeset	96 except IOError as err:
7755 fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6519 diff changeset	97 sys.stdout.write("I/O error: %s\n" % str(err))
fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6519 diff changeset	98 sys.exit(1)
16514 90c600a819e9 String method conversion. Eric S. Raymond <esr@thyrsus.com> parents: 14165 diff changeset	99 lines = fp.read().split("\n")
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	100 fp.close()
6519 24da035ceb6e Convert all remaining simple cases of regex usage to re usage. Guido van Rossum <guido@python.org> parents: 6406 diff changeset	101 prog = re.compile(
25537 61d624f5c107 Updates to track Grammar changes. The patch to token.py loosens the regexp to Michael W. Hudson <mwh@python.net> parents: 19229 diff changeset	102 "#define[ \t][ \t]([A-Z0-9][A-Z0-9_])[ \t][ \t]([0-9][0-9])",
7755 fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6519 diff changeset	103 re.IGNORECASE)
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	104 tokens = {}
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	105 for line in lines:
7755 fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6519 diff changeset	106 match = prog.match(line)
fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6519 diff changeset	107 if match:
fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6519 diff changeset	108 name, val = match.group(1, 2)
16548 da8681f17b50 String method cleanup. Eric S. Raymond <esr@thyrsus.com> parents: 16514 diff changeset	109 val = int(val)
7755 fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6519 diff changeset	110 tokens[val] = name # reverse so we can sort them...
40928 7bc7f740f69f Use better idiom to sort keys. Guido van Rossum <guido@python.org> parents: 40921 diff changeset	111 keys = sorted(tokens.keys())
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	112 # load the output skeleton from the target:
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	113 try:
7755 fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6519 diff changeset	114 fp = open(outFileName)
40587 6b18a095c2e7 SF patch 1631942 by Collin Winter: Guido van Rossum <guido@python.org> parents: 40529 diff changeset	115 except IOError as err:
7755 fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6519 diff changeset	116 sys.stderr.write("I/O error: %s\n" % str(err))
fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6519 diff changeset	117 sys.exit(2)
16514 90c600a819e9 String method conversion. Eric S. Raymond <esr@thyrsus.com> parents: 14165 diff changeset	118 format = fp.read().split("\n")
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	119 fp.close()
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	120 try:
7755 fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6519 diff changeset	121 start = format.index("#--start constants--") + 1
fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6519 diff changeset	122 end = format.index("#--end constants--")
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	123 except ValueError:
7755 fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6519 diff changeset	124 sys.stderr.write("target does not contain format markers")
fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6519 diff changeset	125 sys.exit(3)
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	126 lines = []
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	127 for val in keys:
7755 fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6519 diff changeset	128 lines.append("%s = %d" % (tokens[val], val))
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	129 format[start:end] = lines
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	130 try:
7755 fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6519 diff changeset	131 fp = open(outFileName, 'w')
40587 6b18a095c2e7 SF patch 1631942 by Collin Winter: Guido van Rossum <guido@python.org> parents: 40529 diff changeset	132 except IOError as err:
7755 fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6519 diff changeset	133 sys.stderr.write("I/O error: %s\n" % str(err))
fb4b4d069cc7 Mass check-in after untabifying all files that need it. Guido van Rossum <guido@python.org> parents: 6519 diff changeset	134 sys.exit(4)
16514 90c600a819e9 String method conversion. Eric S. Raymond <esr@thyrsus.com> parents: 14165 diff changeset	135 fp.write("\n".join(format))
4084 7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	136 fp.close()
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	137
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	138
7f83e24d1d86 New batch from Fred Guido van Rossum <guido@python.org> parents: 3754 diff changeset	139 if __name__ == "__main__":
66148 98bd3ca1ed86 Issue #10386: Added __all__ to token module; this simplifies importing Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 59522 diff changeset	140 _main()

Mercurial > cpython-lru_cache-weakref

annotate Lib/token.py @ 70817:dd775b1561ba