cpython-lru_cache-weakref: Tools/unicode/gencodec.py annotate

annotate Tools/unicode/gencodec.py @ 58765:2ba7677bfa95 legacy-trunk

testCopysign was defined twice in test_math; combine the definitions

author	Mark Dickinson <dickinsm@gmail.com>
date	Sat, 06 Feb 2010 23:11:25 +0000
parents	1d84bf5f9478
children	8aa5530e8407

rev	line source
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	1 """ Unicode Mapping Parser and Codec Generator.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	2
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	3 This script parses Unicode mapping files as available from the Unicode
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	4 site (ftp://ftp.unicode.org/Public/MAPPINGS/) and creates Python codec
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	5 modules from them. The codecs use the standard character mapping codec
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	6 to actually apply the mapping.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	7
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	8 Synopsis: gencodec.py dir codec_prefix
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	9
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	10 All files in dir are scanned and those producing non-empty mappings
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	11 will be written to <codec_prefix><mapname>.py with <mapname> being the
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	12 first part of the map's filename ('a' in a.b.c.txt) converted to
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	13 lowercase with hyphens replaced by underscores.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	14
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	15 The tool also writes marshalled versions of the mapping tables to the
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	16 same location (with .mapping extension).
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	17
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	18 Written by Marc-Andre Lemburg (mal@lemburg.com).
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	19
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	20 (c) Copyright CNRI, All Rights Reserved. NO WARRANTY.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	21 (c) Copyright Guido van Rossum, 2000.
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	22
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	23 Table generation:
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	24 (c) Copyright Marc-Andre Lemburg, 2005.
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	25 Licensed to PSF under a Contributor Agreement.
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	26
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	27 """#"
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	28
45468 1d84bf5f9478 Patch #2167 from calvin: Remove unused imports Christian Heimes <christian@cheimes.de> parents: 38586 diff changeset	29 import re, os, marshal, codecs
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	30
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	31 # Maximum allowed size of charmap tables
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	32 MAX_TABLE_SIZE = 8192
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	33
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	34 # Standard undefined Unicode code point
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	35 UNI_UNDEFINED = unichr(0xFFFE)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	36
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	37 mapRE = re.compile('((?:0x[0-9a-fA-F]+\+?)+)'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	38 '\s+'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	39 '((?:(?:0x[0-9a-fA-Z]+\|<[A-Za-z]+>)\+?)*)'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	40 '\s*'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	41 '(#.+)?')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	42
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	43 def parsecodes(codes,
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	44 len=len, filter=filter,range=range):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	45
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	46 """ Converts code combinations to either a single code integer
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	47 or a tuple of integers.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	48
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	49 meta-codes (in angular brackets, e.g. <LR> and <RL>) are
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	50 ignored.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	51
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	52 Empty codes or illegal ones are returned as None.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	53
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	54 """
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	55 if not codes:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	56 return None
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	57 l = codes.split('+')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	58 if len(l) == 1:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	59 return int(l[0],16)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	60 for i in range(len(l)):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	61 try:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	62 l[i] = int(l[i],16)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	63 except ValueError:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	64 l[i] = None
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	65 l = filter(lambda x: x is not None, l)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	66 if len(l) == 1:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	67 return l[0]
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	68 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	69 return tuple(l)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	70
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	71 def readmap(filename):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	72
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	73 f = open(filename,'r')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	74 lines = f.readlines()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	75 f.close()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	76 enc2uni = {}
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	77 identity = []
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	78 unmapped = range(256)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	79
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	80 # UTC mapping tables per convention don't include the identity
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	81 # mappings for code points 0x00 - 0x1F and 0x7F, unless these are
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	82 # explicitly mapped to different characters or undefined
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	83 for i in range(32) + [127]:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	84 identity.append(i)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	85 unmapped.remove(i)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	86 enc2uni[i] = (i, 'CONTROL CHARACTER')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	87
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	88 for line in lines:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	89 line = line.strip()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	90 if not line or line[0] == '#':
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	91 continue
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	92 m = mapRE.match(line)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	93 if not m:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	94 #print '* not matched: %s' % repr(line)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	95 continue
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	96 enc,uni,comment = m.groups()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	97 enc = parsecodes(enc)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	98 uni = parsecodes(uni)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	99 if comment is None:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	100 comment = ''
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	101 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	102 comment = comment[1:].strip()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	103 if enc < 256:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	104 if enc in unmapped:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	105 unmapped.remove(enc)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	106 if enc == uni:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	107 identity.append(enc)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	108 enc2uni[enc] = (uni,comment)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	109 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	110 enc2uni[enc] = (uni,comment)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	111
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	112 # If there are more identity-mapped entries than unmapped entries,
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	113 # it pays to generate an identity dictionary first, and add explicit
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	114 # mappings to None for the rest
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	115 if len(identity) >= len(unmapped):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	116 for enc in unmapped:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	117 enc2uni[enc] = (None, "")
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	118 enc2uni['IDENTITY'] = 256
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	119
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	120 return enc2uni
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	121
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	122 def hexrepr(t, precision=4):
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	123
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	124 if t is None:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	125 return 'None'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	126 try:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	127 len(t)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	128 except:
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	129 return '0x%0*X' % (precision, t)
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	130 try:
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	131 return '(' + ', '.join(['0x%0*X' % (precision, item)
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	132 for item in t]) + ')'
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	133 except TypeError, why:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	134 print '* failed to convert %r: %s' % (t, why)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	135 raise
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	136
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	137 def python_mapdef_code(varname, map, comments=1, precisions=(2, 4)):
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	138
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	139 l = []
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	140 append = l.append
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	141 if map.has_key("IDENTITY"):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	142 append("%s = codecs.make_identity_dict(range(%d))" %
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	143 (varname, map["IDENTITY"]))
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	144 append("%s.update({" % varname)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	145 splits = 1
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	146 del map["IDENTITY"]
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	147 identity = 1
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	148 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	149 append("%s = {" % varname)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	150 splits = 0
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	151 identity = 0
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	152
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	153 mappings = map.items()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	154 mappings.sort()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	155 i = 0
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	156 key_precision, value_precision = precisions
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	157 for mapkey, mapvalue in mappings:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	158 mapcomment = ''
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	159 if isinstance(mapkey, tuple):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	160 (mapkey, mapcomment) = mapkey
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	161 if isinstance(mapvalue, tuple):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	162 (mapvalue, mapcomment) = mapvalue
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	163 if mapkey is None:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	164 continue
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	165 if (identity and
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	166 mapkey == mapvalue and
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	167 mapkey < 256):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	168 # No need to include identity mappings, since these
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	169 # are already set for the first 256 code points.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	170 continue
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	171 key = hexrepr(mapkey, key_precision)
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	172 value = hexrepr(mapvalue, value_precision)
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	173 if mapcomment and comments:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	174 append(' %s: %s,\t# %s' % (key, value, mapcomment))
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	175 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	176 append(' %s: %s,' % (key, value))
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	177 i += 1
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	178 if i == 4096:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	179 # Split the definition into parts to that the Python
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	180 # parser doesn't dump core
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	181 if splits == 0:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	182 append('}')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	183 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	184 append('})')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	185 append('%s.update({' % varname)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	186 i = 0
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	187 splits = splits + 1
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	188 if splits == 0:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	189 append('}')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	190 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	191 append('})')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	192
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	193 return l
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	194
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	195 def python_tabledef_code(varname, map, comments=1, key_precision=2):
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	196
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	197 l = []
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	198 append = l.append
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	199 append('%s = (' % varname)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	200
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	201 # Analyze map and create table dict
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	202 mappings = map.items()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	203 mappings.sort()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	204 table = {}
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	205 maxkey = 0
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	206 if map.has_key('IDENTITY'):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	207 for key in range(256):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	208 table[key] = (key, '')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	209 maxkey = 255
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	210 del map['IDENTITY']
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	211 for mapkey, mapvalue in mappings:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	212 mapcomment = ''
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	213 if isinstance(mapkey, tuple):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	214 (mapkey, mapcomment) = mapkey
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	215 if isinstance(mapvalue, tuple):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	216 (mapvalue, mapcomment) = mapvalue
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	217 if mapkey is None:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	218 continue
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	219 table[mapkey] = (mapvalue, mapcomment)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	220 if mapkey > maxkey:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	221 maxkey = mapkey
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	222 if maxkey > MAX_TABLE_SIZE:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	223 # Table too large
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	224 return None
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	225
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	226 # Create table code
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	227 for key in range(maxkey + 1):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	228 if key not in table:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	229 mapvalue = None
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	230 mapcomment = 'UNDEFINED'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	231 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	232 mapvalue, mapcomment = table[key]
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	233 if mapvalue is None:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	234 mapchar = UNI_UNDEFINED
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	235 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	236 if isinstance(mapvalue, tuple):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	237 # 1-n mappings not supported
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	238 return None
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	239 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	240 mapchar = unichr(mapvalue)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	241 if mapcomment and comments:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	242 append(' %r\t# %s -> %s' % (mapchar,
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	243 hexrepr(key, key_precision),
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	244 mapcomment))
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	245 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	246 append(' %r' % mapchar)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	247
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	248 append(')')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	249 return l
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	250
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	251 def codegen(name, map, encodingname, comments=1):
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	252
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	253 """ Returns Python source for the given map.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	254
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	255 Comments are included in the source, if comments is true (default).
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	256
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	257 """
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	258 # Generate code
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	259 decoding_map_code = python_mapdef_code(
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	260 'decoding_map',
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	261 map,
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	262 comments=comments)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	263 decoding_table_code = python_tabledef_code(
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	264 'decoding_table',
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	265 map,
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	266 comments=comments)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	267 encoding_map_code = python_mapdef_code(
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	268 'encoding_map',
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	269 codecs.make_encoding_map(map),
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	270 comments=comments,
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	271 precisions=(4, 2))
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	272
38586 9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	273 if decoding_table_code:
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	274 suffix = 'table'
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	275 else:
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	276 suffix = 'map'
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	277
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	278 l = [
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	279 '''\
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	280 """ Python Character Mapping Codec %s generated from '%s' with gencodec.py.
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	281
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	282 """#"
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	283
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	284 import codecs
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	285
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	286 ### Codec APIs
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	287
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	288 class Codec(codecs.Codec):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	289
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	290 def encode(self,input,errors='strict'):
38586 9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	291 return codecs.charmap_encode(input,errors,encoding_%s)
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	292
38586 9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	293 def decode(self,input,errors='strict'):
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	294 return codecs.charmap_decode(input,errors,decoding_%s)
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	295 ''' % (encodingname, name, suffix, suffix)]
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	296 l.append('''\
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	297 class IncrementalEncoder(codecs.IncrementalEncoder):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	298 def encode(self, input, final=False):
38586 9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	299 return codecs.charmap_encode(input,self.errors,encoding_%s)[0]
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	300
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	301 class IncrementalDecoder(codecs.IncrementalDecoder):
38586 9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	302 def decode(self, input, final=False):
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	303 return codecs.charmap_decode(input,self.errors,decoding_%s)[0]''' %
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	304 (suffix, suffix))
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	305
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	306 l.append('''
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	307 class StreamWriter(Codec,codecs.StreamWriter):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	308 pass
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	309
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	310 class StreamReader(Codec,codecs.StreamReader):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	311 pass
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	312
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	313 ### encodings module API
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	314
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	315 def getregentry():
38346 92b0fc54d230 when generating python code prefer to generate valid python code Jack Diederich <jackdied@gmail.com> parents: 37323 diff changeset	316 return codecs.CodecInfo(
92b0fc54d230 when generating python code prefer to generate valid python code Jack Diederich <jackdied@gmail.com> parents: 37323 diff changeset	317 name=%r,
38586 9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	318 encode=Codec().encode,
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	319 decode=Codec().decode,
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	320 incrementalencoder=IncrementalEncoder,
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	321 incrementaldecoder=IncrementalDecoder,
38586 9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	322 streamreader=StreamReader,
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	323 streamwriter=StreamWriter,
38346 92b0fc54d230 when generating python code prefer to generate valid python code Jack Diederich <jackdied@gmail.com> parents: 37323 diff changeset	324 )
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	325 ''' % encodingname.replace('_', '-'))
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	326
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	327 # Add decoding table or map (with preference to the table)
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	328 if not decoding_table_code:
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	329 l.append('''
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	330 ### Decoding Map
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	331 ''')
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	332 l.extend(decoding_map_code)
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	333 else:
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	334 l.append('''
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	335 ### Decoding Table
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	336 ''')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	337 l.extend(decoding_table_code)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	338
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	339 # Add encoding map
38586 9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	340 if decoding_table_code:
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	341 l.append('''
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	342 ### Encoding table
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	343 encoding_table=codecs.charmap_build(decoding_table)
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	344 ''')
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	345 else:
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	346 l.append('''
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	347 ### Encoding Map
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	348 ''')
38586 9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	349 l.extend(encoding_map_code)
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	350
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	351 # Final new-line
37323 164f95e8e0cc Don't add multiple empty lines at the end of the codec. With this a Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	352 l.append('')
36121 686306a67e9b Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 35800 diff changeset	353
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	354 return '\n'.join(l).expandtabs()
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	355
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	356 def pymap(name,map,pyfile,encodingname,comments=1):
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	357
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	358 code = codegen(name,map,encodingname,comments)
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	359 f = open(pyfile,'w')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	360 f.write(code)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	361 f.close()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	362
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	363 def marshalmap(name,map,marshalfile):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	364
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	365 d = {}
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	366 for e,(u,c) in map.items():
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	367 d[e] = (u,c)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	368 f = open(marshalfile,'wb')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	369 marshal.dump(d,f)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	370 f.close()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	371
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	372 def convertdir(dir, dirprefix='', nameprefix='', comments=1):
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	373
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	374 mapnames = os.listdir(dir)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	375 for mapname in mapnames:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	376 mappathname = os.path.join(dir, mapname)
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	377 if not os.path.isfile(mappathname):
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	378 continue
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	379 name = os.path.split(mapname)[1]
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	380 name = name.replace('-','_')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	381 name = name.split('.')[0]
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	382 name = name.lower()
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	383 name = nameprefix + name
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	384 codefile = name + '.py'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	385 marshalfile = name + '.mapping'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	386 print 'converting %s to %s and %s' % (mapname,
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	387 dirprefix + codefile,
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	388 dirprefix + marshalfile)
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	389 try:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	390 map = readmap(os.path.join(dir,mapname))
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	391 if not map:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	392 print '* map is empty; skipping'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	393 else:
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	394 pymap(mappathname, map, dirprefix + codefile,name,comments)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	395 marshalmap(mappathname, map, dirprefix + marshalfile)
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	396 except ValueError, why:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	397 print '* conversion failed: %s' % why
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	398 raise
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	399
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	400 def rewritepythondir(dir, dirprefix='', comments=1):
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	401
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	402 mapnames = os.listdir(dir)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	403 for mapname in mapnames:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	404 if not mapname.endswith('.mapping'):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	405 continue
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	406 name = mapname[:-len('.mapping')]
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	407 codefile = name + '.py'
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	408 print 'converting %s to %s' % (mapname,
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	409 dirprefix + codefile)
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	410 try:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	411 map = marshal.load(open(os.path.join(dir,mapname),
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	412 'rb'))
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	413 if not map:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	414 print '* map is empty; skipping'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	415 else:
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	416 pymap(mapname, map, dirprefix + codefile,name,comments)
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	417 except ValueError, why:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	418 print '* conversion failed: %s' % why
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	419
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	420 if __name__ == '__main__':
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	421
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	422 import sys
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	423 if 1:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	424 apply(convertdir,tuple(sys.argv[1:]))
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	425 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	426 apply(rewritepythondir,tuple(sys.argv[1:]))

Mercurial > cpython-lru_cache-weakref

annotate Tools/unicode/gencodec.py @ 58765:2ba7677bfa95 legacy-trunk