cpython-lru_cache-weakref: Tools/unicode/gencodec.py annotate

annotate Tools/unicode/gencodec.py @ 74135:9c19df6c8ea0 3.2

News entry for Issue12798

author	Senthil Kumaran <senthil@uthcode.com>
date	Fri, 23 Dec 2011 11:11:21 +0800
parents	f79928adab57
children	8f3a5308f50b

rev	line source
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	1 """ Unicode Mapping Parser and Codec Generator.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	2
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	3 This script parses Unicode mapping files as available from the Unicode
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	4 site (ftp://ftp.unicode.org/Public/MAPPINGS/) and creates Python codec
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	5 modules from them. The codecs use the standard character mapping codec
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	6 to actually apply the mapping.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	7
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	8 Synopsis: gencodec.py dir codec_prefix
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	9
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	10 All files in dir are scanned and those producing non-empty mappings
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	11 will be written to <codec_prefix><mapname>.py with <mapname> being the
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	12 first part of the map's filename ('a' in a.b.c.txt) converted to
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	13 lowercase with hyphens replaced by underscores.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	14
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	15 The tool also writes marshalled versions of the mapping tables to the
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	16 same location (with .mapping extension).
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	17
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	18 Written by Marc-Andre Lemburg (mal@lemburg.com).
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	19
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	20 (c) Copyright CNRI, All Rights Reserved. NO WARRANTY.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	21 (c) Copyright Guido van Rossum, 2000.
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	22
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	23 Table generation:
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	24 (c) Copyright Marc-Andre Lemburg, 2005.
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	25 Licensed to PSF under a Contributor Agreement.
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	26
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	27 """#"
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	28
45471 7dd2c1746267 Merged revisions 60990-61002 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 42427 diff changeset	29 import re, os, marshal, codecs
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	30
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	31 # Maximum allowed size of charmap tables
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	32 MAX_TABLE_SIZE = 8192
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	33
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	34 # Standard undefined Unicode code point
47103 63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893. Georg Brandl <georg@python.org> parents: 45471 diff changeset	35 UNI_UNDEFINED = chr(0xFFFE)
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	36
66563 f79928adab57 Issue #10552: Partially fixed a sort error in Tools/unicode/gencodec.py Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 59720 diff changeset	37 # Placeholder for a missing codepoint
f79928adab57 Issue #10552: Partially fixed a sort error in Tools/unicode/gencodec.py Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 59720 diff changeset	38 MISSING_CODE = -1
f79928adab57 Issue #10552: Partially fixed a sort error in Tools/unicode/gencodec.py Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 59720 diff changeset	39
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	40 mapRE = re.compile('((?:0x[0-9a-fA-F]+\+?)+)'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	41 '\s+'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	42 '((?:(?:0x[0-9a-fA-Z]+\|<[A-Za-z]+>)\+?)*)'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	43 '\s*'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	44 '(#.+)?')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	45
59720 8a28be45c242 Merged revisions 78982,78986 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 55249 diff changeset	46 def parsecodes(codes, len=len, range=range):
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	47
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	48 """ Converts code combinations to either a single code integer
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	49 or a tuple of integers.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	50
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	51 meta-codes (in angular brackets, e.g. <LR> and <RL>) are
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	52 ignored.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	53
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	54 Empty codes or illegal ones are returned as None.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	55
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	56 """
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	57 if not codes:
66563 f79928adab57 Issue #10552: Partially fixed a sort error in Tools/unicode/gencodec.py Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 59720 diff changeset	58 return MISSING_CODE
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	59 l = codes.split('+')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	60 if len(l) == 1:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	61 return int(l[0],16)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	62 for i in range(len(l)):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	63 try:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	64 l[i] = int(l[i],16)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	65 except ValueError:
66563 f79928adab57 Issue #10552: Partially fixed a sort error in Tools/unicode/gencodec.py Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 59720 diff changeset	66 l[i] = MISSING_CODE
f79928adab57 Issue #10552: Partially fixed a sort error in Tools/unicode/gencodec.py Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 59720 diff changeset	67 l = [x for x in l if x != MISSING_CODE]
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	68 if len(l) == 1:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	69 return l[0]
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	70 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	71 return tuple(l)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	72
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	73 def readmap(filename):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	74
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	75 f = open(filename,'r')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	76 lines = f.readlines()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	77 f.close()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	78 enc2uni = {}
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	79 identity = []
47103 63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893. Georg Brandl <georg@python.org> parents: 45471 diff changeset	80 unmapped = list(range(256))
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	81
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	82 # UTC mapping tables per convention don't include the identity
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	83 # mappings for code points 0x00 - 0x1F and 0x7F, unless these are
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	84 # explicitly mapped to different characters or undefined
47103 63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893. Georg Brandl <georg@python.org> parents: 45471 diff changeset	85 for i in list(range(32)) + [127]:
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	86 identity.append(i)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	87 unmapped.remove(i)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	88 enc2uni[i] = (i, 'CONTROL CHARACTER')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	89
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	90 for line in lines:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	91 line = line.strip()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	92 if not line or line[0] == '#':
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	93 continue
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	94 m = mapRE.match(line)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	95 if not m:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	96 #print '* not matched: %s' % repr(line)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	97 continue
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	98 enc,uni,comment = m.groups()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	99 enc = parsecodes(enc)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	100 uni = parsecodes(uni)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	101 if comment is None:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	102 comment = ''
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	103 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	104 comment = comment[1:].strip()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	105 if enc < 256:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	106 if enc in unmapped:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	107 unmapped.remove(enc)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	108 if enc == uni:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	109 identity.append(enc)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	110 enc2uni[enc] = (uni,comment)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	111 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	112 enc2uni[enc] = (uni,comment)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	113
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	114 # If there are more identity-mapped entries than unmapped entries,
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	115 # it pays to generate an identity dictionary first, and add explicit
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	116 # mappings to None for the rest
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	117 if len(identity) >= len(unmapped):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	118 for enc in unmapped:
66563 f79928adab57 Issue #10552: Partially fixed a sort error in Tools/unicode/gencodec.py Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 59720 diff changeset	119 enc2uni[enc] = (MISSING_CODE, "")
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	120 enc2uni['IDENTITY'] = 256
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	121
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	122 return enc2uni
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	123
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	124 def hexrepr(t, precision=4):
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	125
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	126 if t is None:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	127 return 'None'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	128 try:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	129 len(t)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	130 except:
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	131 return '0x%0*X' % (precision, t)
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	132 try:
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	133 return '(' + ', '.join(['0x%0*X' % (precision, item)
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	134 for item in t]) + ')'
40587 6b18a095c2e7 SF patch 1631942 by Collin Winter: Guido van Rossum <guido@python.org> parents: 38673 diff changeset	135 except TypeError as why:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 40587 diff changeset	136 print('* failed to convert %r: %s' % (t, why))
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	137 raise
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	138
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	139 def python_mapdef_code(varname, map, comments=1, precisions=(2, 4)):
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	140
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	141 l = []
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	142 append = l.append
47103 63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893. Georg Brandl <georg@python.org> parents: 45471 diff changeset	143 if "IDENTITY" in map:
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	144 append("%s = codecs.make_identity_dict(range(%d))" %
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	145 (varname, map["IDENTITY"]))
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	146 append("%s.update({" % varname)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	147 splits = 1
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	148 del map["IDENTITY"]
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	149 identity = 1
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	150 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	151 append("%s = {" % varname)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	152 splits = 0
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	153 identity = 0
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	154
47103 63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893. Georg Brandl <georg@python.org> parents: 45471 diff changeset	155 mappings = sorted(map.items())
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	156 i = 0
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	157 key_precision, value_precision = precisions
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	158 for mapkey, mapvalue in mappings:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	159 mapcomment = ''
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	160 if isinstance(mapkey, tuple):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	161 (mapkey, mapcomment) = mapkey
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	162 if isinstance(mapvalue, tuple):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	163 (mapvalue, mapcomment) = mapvalue
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	164 if mapkey is None:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	165 continue
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	166 if (identity and
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	167 mapkey == mapvalue and
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	168 mapkey < 256):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	169 # No need to include identity mappings, since these
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	170 # are already set for the first 256 code points.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	171 continue
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	172 key = hexrepr(mapkey, key_precision)
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	173 value = hexrepr(mapvalue, value_precision)
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	174 if mapcomment and comments:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	175 append(' %s: %s,\t# %s' % (key, value, mapcomment))
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	176 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	177 append(' %s: %s,' % (key, value))
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	178 i += 1
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	179 if i == 4096:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	180 # Split the definition into parts to that the Python
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	181 # parser doesn't dump core
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	182 if splits == 0:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	183 append('}')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	184 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	185 append('})')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	186 append('%s.update({' % varname)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	187 i = 0
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	188 splits = splits + 1
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	189 if splits == 0:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	190 append('}')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	191 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	192 append('})')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	193
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	194 return l
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	195
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	196 def python_tabledef_code(varname, map, comments=1, key_precision=2):
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	197
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	198 l = []
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	199 append = l.append
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	200 append('%s = (' % varname)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	201
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	202 # Analyze map and create table dict
47103 63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893. Georg Brandl <georg@python.org> parents: 45471 diff changeset	203 mappings = sorted(map.items())
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	204 table = {}
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	205 maxkey = 0
47103 63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893. Georg Brandl <georg@python.org> parents: 45471 diff changeset	206 if 'IDENTITY' in map:
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	207 for key in range(256):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	208 table[key] = (key, '')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	209 maxkey = 255
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	210 del map['IDENTITY']
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	211 for mapkey, mapvalue in mappings:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	212 mapcomment = ''
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	213 if isinstance(mapkey, tuple):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	214 (mapkey, mapcomment) = mapkey
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	215 if isinstance(mapvalue, tuple):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	216 (mapvalue, mapcomment) = mapvalue
66563 f79928adab57 Issue #10552: Partially fixed a sort error in Tools/unicode/gencodec.py Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 59720 diff changeset	217 if mapkey == MISSING_CODE:
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	218 continue
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	219 table[mapkey] = (mapvalue, mapcomment)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	220 if mapkey > maxkey:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	221 maxkey = mapkey
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	222 if maxkey > MAX_TABLE_SIZE:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	223 # Table too large
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	224 return None
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	225
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	226 # Create table code
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	227 for key in range(maxkey + 1):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	228 if key not in table:
66563 f79928adab57 Issue #10552: Partially fixed a sort error in Tools/unicode/gencodec.py Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 59720 diff changeset	229 mapvalue = MISSING_CODE
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	230 mapcomment = 'UNDEFINED'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	231 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	232 mapvalue, mapcomment = table[key]
66563 f79928adab57 Issue #10552: Partially fixed a sort error in Tools/unicode/gencodec.py Alexander Belopolsky <alexander.belopolsky@gmail.com> parents: 59720 diff changeset	233 if mapvalue == MISSING_CODE:
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	234 mapchar = UNI_UNDEFINED
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	235 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	236 if isinstance(mapvalue, tuple):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	237 # 1-n mappings not supported
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	238 return None
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	239 else:
47103 63741c7521aa More 2to3 fixes in the Tools directory. Fixes #2893. Georg Brandl <georg@python.org> parents: 45471 diff changeset	240 mapchar = chr(mapvalue)
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	241 if mapcomment and comments:
55249 ef1b1198338b Merged revisions 74000-74001 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 47103 diff changeset	242 append(' %a \t# %s -> %s' % (mapchar,
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	243 hexrepr(key, key_precision),
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	244 mapcomment))
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	245 else:
55249 ef1b1198338b Merged revisions 74000-74001 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 47103 diff changeset	246 append(' %a' % mapchar)
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	247
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	248 append(')')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	249 return l
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	250
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 37048 diff changeset	251 def codegen(name, map, encodingname, comments=1):
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	252
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	253 """ Returns Python source for the given map.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	254
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	255 Comments are included in the source, if comments is true (default).
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	256
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	257 """
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	258 # Generate code
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	259 decoding_map_code = python_mapdef_code(
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	260 'decoding_map',
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	261 map,
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	262 comments=comments)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	263 decoding_table_code = python_tabledef_code(
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	264 'decoding_table',
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	265 map,
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	266 comments=comments)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	267 encoding_map_code = python_mapdef_code(
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	268 'encoding_map',
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	269 codecs.make_encoding_map(map),
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	270 comments=comments,
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	271 precisions=(4, 2))
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	272
38673 a19c20c5e5c7 Merge the rest of the trunk. Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	273 if decoding_table_code:
a19c20c5e5c7 Merge the rest of the trunk. Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	274 suffix = 'table'
a19c20c5e5c7 Merge the rest of the trunk. Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	275 else:
a19c20c5e5c7 Merge the rest of the trunk. Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	276 suffix = 'map'
a19c20c5e5c7 Merge the rest of the trunk. Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	277
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	278 l = [
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	279 '''\
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 37048 diff changeset	280 """ Python Character Mapping Codec %s generated from '%s' with gencodec.py.
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	281
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	282 """#"
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	283
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	284 import codecs
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	285
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	286 ### Codec APIs
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	287
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	288 class Codec(codecs.Codec):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	289
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	290 def encode(self,input,errors='strict'):
38673 a19c20c5e5c7 Merge the rest of the trunk. Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	291 return codecs.charmap_encode(input,errors,encoding_%s)
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	292
38673 a19c20c5e5c7 Merge the rest of the trunk. Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	293 def decode(self,input,errors='strict'):
a19c20c5e5c7 Merge the rest of the trunk. Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	294 return codecs.charmap_decode(input,errors,decoding_%s)
a19c20c5e5c7 Merge the rest of the trunk. Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	295 ''' % (encodingname, name, suffix, suffix)]
a19c20c5e5c7 Merge the rest of the trunk. Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	296 l.append('''\
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 37048 diff changeset	297 class IncrementalEncoder(codecs.IncrementalEncoder):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 37048 diff changeset	298 def encode(self, input, final=False):
38673 a19c20c5e5c7 Merge the rest of the trunk. Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	299 return codecs.charmap_encode(input,self.errors,encoding_%s)[0]
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 37048 diff changeset	300
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 37048 diff changeset	301 class IncrementalDecoder(codecs.IncrementalDecoder):
38673 a19c20c5e5c7 Merge the rest of the trunk. Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	302 def decode(self, input, final=False):
a19c20c5e5c7 Merge the rest of the trunk. Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	303 return codecs.charmap_decode(input,self.errors,decoding_%s)[0]''' %
a19c20c5e5c7 Merge the rest of the trunk. Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	304 (suffix, suffix))
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 37048 diff changeset	305
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 37048 diff changeset	306 l.append('''
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	307 class StreamWriter(Codec,codecs.StreamWriter):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	308 pass
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	309
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	310 class StreamReader(Codec,codecs.StreamReader):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	311 pass
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	312
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	313 ### encodings module API
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	314
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	315 def getregentry():
38453 29ae5e0fc348 Much-needed merge (using svnmerge.py this time) of trunk changes into p3yk. Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	316 return codecs.CodecInfo(
29ae5e0fc348 Much-needed merge (using svnmerge.py this time) of trunk changes into p3yk. Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	317 name=%r,
38673 a19c20c5e5c7 Merge the rest of the trunk. Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	318 encode=Codec().encode,
a19c20c5e5c7 Merge the rest of the trunk. Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	319 decode=Codec().decode,
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 37048 diff changeset	320 incrementalencoder=IncrementalEncoder,
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 37048 diff changeset	321 incrementaldecoder=IncrementalDecoder,
38673 a19c20c5e5c7 Merge the rest of the trunk. Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	322 streamreader=StreamReader,
a19c20c5e5c7 Merge the rest of the trunk. Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	323 streamwriter=StreamWriter,
38453 29ae5e0fc348 Much-needed merge (using svnmerge.py this time) of trunk changes into p3yk. Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	324 )
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 37048 diff changeset	325 ''' % encodingname.replace('_', '-'))
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	326
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	327 # Add decoding table or map (with preference to the table)
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	328 if not decoding_table_code:
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	329 l.append('''
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	330 ### Decoding Map
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	331 ''')
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	332 l.extend(decoding_map_code)
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	333 else:
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	334 l.append('''
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	335 ### Decoding Table
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	336 ''')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	337 l.extend(decoding_table_code)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	338
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	339 # Add encoding map
38673 a19c20c5e5c7 Merge the rest of the trunk. Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	340 if decoding_table_code:
a19c20c5e5c7 Merge the rest of the trunk. Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	341 l.append('''
a19c20c5e5c7 Merge the rest of the trunk. Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	342 ### Encoding table
a19c20c5e5c7 Merge the rest of the trunk. Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	343 encoding_table=codecs.charmap_build(decoding_table)
a19c20c5e5c7 Merge the rest of the trunk. Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	344 ''')
a19c20c5e5c7 Merge the rest of the trunk. Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	345 else:
a19c20c5e5c7 Merge the rest of the trunk. Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	346 l.append('''
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	347 ### Encoding Map
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	348 ''')
38673 a19c20c5e5c7 Merge the rest of the trunk. Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	349 l.extend(encoding_map_code)
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	350
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	351 # Final new-line
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	352 l.append('')
36121 686306a67e9b Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 35800 diff changeset	353
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 37048 diff changeset	354 return '\n'.join(l).expandtabs()
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	355
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 37048 diff changeset	356 def pymap(name,map,pyfile,encodingname,comments=1):
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	357
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 37048 diff changeset	358 code = codegen(name,map,encodingname,comments)
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	359 f = open(pyfile,'w')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	360 f.write(code)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	361 f.close()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	362
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	363 def marshalmap(name,map,marshalfile):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	364
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	365 d = {}
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	366 for e,(u,c) in map.items():
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	367 d[e] = (u,c)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	368 f = open(marshalfile,'wb')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	369 marshal.dump(d,f)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	370 f.close()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	371
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 37048 diff changeset	372 def convertdir(dir, dirprefix='', nameprefix='', comments=1):
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	373
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	374 mapnames = os.listdir(dir)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	375 for mapname in mapnames:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	376 mappathname = os.path.join(dir, mapname)
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	377 if not os.path.isfile(mappathname):
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	378 continue
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	379 name = os.path.split(mapname)[1]
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	380 name = name.replace('-','_')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	381 name = name.split('.')[0]
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	382 name = name.lower()
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 37048 diff changeset	383 name = nameprefix + name
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	384 codefile = name + '.py'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	385 marshalfile = name + '.mapping'
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 40587 diff changeset	386 print('converting %s to %s and %s' % (mapname,
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 37048 diff changeset	387 dirprefix + codefile,
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 40587 diff changeset	388 dirprefix + marshalfile))
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	389 try:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	390 map = readmap(os.path.join(dir,mapname))
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	391 if not map:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 40587 diff changeset	392 print('* map is empty; skipping')
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	393 else:
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 37048 diff changeset	394 pymap(mappathname, map, dirprefix + codefile,name,comments)
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 37048 diff changeset	395 marshalmap(mappathname, map, dirprefix + marshalfile)
40587 6b18a095c2e7 SF patch 1631942 by Collin Winter: Guido van Rossum <guido@python.org> parents: 38673 diff changeset	396 except ValueError as why:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 40587 diff changeset	397 print('* conversion failed: %s' % why)
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	398 raise
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	399
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 37048 diff changeset	400 def rewritepythondir(dir, dirprefix='', comments=1):
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	401
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	402 mapnames = os.listdir(dir)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	403 for mapname in mapnames:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	404 if not mapname.endswith('.mapping'):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	405 continue
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 37048 diff changeset	406 name = mapname[:-len('.mapping')]
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 37048 diff changeset	407 codefile = name + '.py'
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 40587 diff changeset	408 print('converting %s to %s' % (mapname,
46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 40587 diff changeset	409 dirprefix + codefile))
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	410 try:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	411 map = marshal.load(open(os.path.join(dir,mapname),
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	412 'rb'))
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	413 if not map:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 40587 diff changeset	414 print('* map is empty; skipping')
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	415 else:
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 37048 diff changeset	416 pymap(mapname, map, dirprefix + codefile,name,comments)
40587 6b18a095c2e7 SF patch 1631942 by Collin Winter: Guido van Rossum <guido@python.org> parents: 38673 diff changeset	417 except ValueError as why:
42427 46625e6122e9 Convert print statements to function calls in Tools/. Collin Winter <collinw@gmail.com> parents: 40587 diff changeset	418 print('* conversion failed: %s' % why)
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	419
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	420 if __name__ == '__main__':
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	421
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	422 import sys
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	423 if 1:
37048 c048a09d06ff Remove apply() Neal Norwitz <nnorwitz@gmail.com> parents: 36121 diff changeset	424 convertdir(*sys.argv[1:])
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	425 else:
37048 c048a09d06ff Remove apply() Neal Norwitz <nnorwitz@gmail.com> parents: 36121 diff changeset	426 rewritepythondir(*sys.argv[1:])

Mercurial > cpython-lru_cache-weakref

annotate Tools/unicode/gencodec.py @ 74135:9c19df6c8ea0 3.2