cpython-lru_cache-weakref: Tools/unicode/gencodec.py annotate

annotate Tools/unicode/gencodec.py @ 37850:ebe59cd18273

Use explicit relative import for an, ehm, relative import.

author	Thomas Wouters <thomas@python.org>
date	Tue, 18 Apr 2006 21:41:36 +0000
parents	c048a09d06ff
children	a4c70f3968be

rev	line source
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	1 """ Unicode Mapping Parser and Codec Generator.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	2
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	3 This script parses Unicode mapping files as available from the Unicode
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	4 site (ftp://ftp.unicode.org/Public/MAPPINGS/) and creates Python codec
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	5 modules from them. The codecs use the standard character mapping codec
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	6 to actually apply the mapping.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	7
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	8 Synopsis: gencodec.py dir codec_prefix
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	9
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	10 All files in dir are scanned and those producing non-empty mappings
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	11 will be written to <codec_prefix><mapname>.py with <mapname> being the
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	12 first part of the map's filename ('a' in a.b.c.txt) converted to
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	13 lowercase with hyphens replaced by underscores.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	14
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	15 The tool also writes marshalled versions of the mapping tables to the
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	16 same location (with .mapping extension).
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	17
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	18 Written by Marc-Andre Lemburg (mal@lemburg.com).
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	19
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	20 (c) Copyright CNRI, All Rights Reserved. NO WARRANTY.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	21 (c) Copyright Guido van Rossum, 2000.
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	22
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	23 Table generation:
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	24 (c) Copyright Marc-Andre Lemburg, 2005.
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	25 Licensed to PSF under a Contributor Agreement.
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	26
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	27 """#"
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	28
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	29 import re, os, time, marshal, codecs
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	30
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	31 # Maximum allowed size of charmap tables
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	32 MAX_TABLE_SIZE = 8192
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	33
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	34 # Standard undefined Unicode code point
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	35 UNI_UNDEFINED = unichr(0xFFFE)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	36
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	37 mapRE = re.compile('((?:0x[0-9a-fA-F]+\+?)+)'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	38 '\s+'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	39 '((?:(?:0x[0-9a-fA-Z]+\|<[A-Za-z]+>)\+?)*)'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	40 '\s*'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	41 '(#.+)?')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	42
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	43 def parsecodes(codes,
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	44 len=len, filter=filter,range=range):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	45
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	46 """ Converts code combinations to either a single code integer
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	47 or a tuple of integers.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	48
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	49 meta-codes (in angular brackets, e.g. <LR> and <RL>) are
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	50 ignored.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	51
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	52 Empty codes or illegal ones are returned as None.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	53
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	54 """
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	55 if not codes:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	56 return None
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	57 l = codes.split('+')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	58 if len(l) == 1:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	59 return int(l[0],16)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	60 for i in range(len(l)):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	61 try:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	62 l[i] = int(l[i],16)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	63 except ValueError:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	64 l[i] = None
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	65 l = filter(lambda x: x is not None, l)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	66 if len(l) == 1:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	67 return l[0]
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	68 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	69 return tuple(l)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	70
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	71 def readmap(filename):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	72
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	73 f = open(filename,'r')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	74 lines = f.readlines()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	75 f.close()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	76 enc2uni = {}
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	77 identity = []
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	78 unmapped = range(256)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	79
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	80 # UTC mapping tables per convention don't include the identity
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	81 # mappings for code points 0x00 - 0x1F and 0x7F, unless these are
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	82 # explicitly mapped to different characters or undefined
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	83 for i in range(32) + [127]:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	84 identity.append(i)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	85 unmapped.remove(i)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	86 enc2uni[i] = (i, 'CONTROL CHARACTER')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	87
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	88 for line in lines:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	89 line = line.strip()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	90 if not line or line[0] == '#':
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	91 continue
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	92 m = mapRE.match(line)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	93 if not m:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	94 #print '* not matched: %s' % repr(line)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	95 continue
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	96 enc,uni,comment = m.groups()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	97 enc = parsecodes(enc)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	98 uni = parsecodes(uni)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	99 if comment is None:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	100 comment = ''
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	101 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	102 comment = comment[1:].strip()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	103 if enc < 256:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	104 if enc in unmapped:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	105 unmapped.remove(enc)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	106 if enc == uni:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	107 identity.append(enc)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	108 enc2uni[enc] = (uni,comment)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	109 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	110 enc2uni[enc] = (uni,comment)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	111
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	112 # If there are more identity-mapped entries than unmapped entries,
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	113 # it pays to generate an identity dictionary first, and add explicit
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	114 # mappings to None for the rest
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	115 if len(identity) >= len(unmapped):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	116 for enc in unmapped:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	117 enc2uni[enc] = (None, "")
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	118 enc2uni['IDENTITY'] = 256
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	119
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	120 return enc2uni
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	121
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	122 def hexrepr(t, precision=4):
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	123
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	124 if t is None:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	125 return 'None'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	126 try:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	127 len(t)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	128 except:
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	129 return '0x%0*X' % (precision, t)
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	130 try:
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	131 return '(' + ', '.join(['0x%0*X' % (precision, item)
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	132 for item in t]) + ')'
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	133 except TypeError, why:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	134 print '* failed to convert %r: %s' % (t, why)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	135 raise
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	136
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	137 def python_mapdef_code(varname, map, comments=1, precisions=(2, 4)):
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	138
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	139 l = []
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	140 append = l.append
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	141 if map.has_key("IDENTITY"):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	142 append("%s = codecs.make_identity_dict(range(%d))" %
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	143 (varname, map["IDENTITY"]))
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	144 append("%s.update({" % varname)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	145 splits = 1
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	146 del map["IDENTITY"]
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	147 identity = 1
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	148 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	149 append("%s = {" % varname)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	150 splits = 0
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	151 identity = 0
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	152
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	153 mappings = map.items()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	154 mappings.sort()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	155 i = 0
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	156 key_precision, value_precision = precisions
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	157 for mapkey, mapvalue in mappings:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	158 mapcomment = ''
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	159 if isinstance(mapkey, tuple):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	160 (mapkey, mapcomment) = mapkey
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	161 if isinstance(mapvalue, tuple):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	162 (mapvalue, mapcomment) = mapvalue
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	163 if mapkey is None:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	164 continue
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	165 if (identity and
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	166 mapkey == mapvalue and
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	167 mapkey < 256):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	168 # No need to include identity mappings, since these
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	169 # are already set for the first 256 code points.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	170 continue
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	171 key = hexrepr(mapkey, key_precision)
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	172 value = hexrepr(mapvalue, value_precision)
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	173 if mapcomment and comments:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	174 append(' %s: %s,\t# %s' % (key, value, mapcomment))
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	175 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	176 append(' %s: %s,' % (key, value))
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	177 i += 1
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	178 if i == 4096:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	179 # Split the definition into parts to that the Python
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	180 # parser doesn't dump core
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	181 if splits == 0:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	182 append('}')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	183 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	184 append('})')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	185 append('%s.update({' % varname)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	186 i = 0
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	187 splits = splits + 1
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	188 if splits == 0:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	189 append('}')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	190 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	191 append('})')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	192
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	193 return l
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	194
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	195 def python_tabledef_code(varname, map, comments=1, key_precision=2):
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	196
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	197 l = []
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	198 append = l.append
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	199 append('%s = (' % varname)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	200
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	201 # Analyze map and create table dict
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	202 mappings = map.items()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	203 mappings.sort()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	204 table = {}
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	205 maxkey = 0
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	206 if map.has_key('IDENTITY'):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	207 for key in range(256):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	208 table[key] = (key, '')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	209 maxkey = 255
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	210 del map['IDENTITY']
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	211 for mapkey, mapvalue in mappings:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	212 mapcomment = ''
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	213 if isinstance(mapkey, tuple):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	214 (mapkey, mapcomment) = mapkey
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	215 if isinstance(mapvalue, tuple):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	216 (mapvalue, mapcomment) = mapvalue
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	217 if mapkey is None:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	218 continue
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	219 table[mapkey] = (mapvalue, mapcomment)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	220 if mapkey > maxkey:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	221 maxkey = mapkey
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	222 if maxkey > MAX_TABLE_SIZE:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	223 # Table too large
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	224 return None
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	225
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	226 # Create table code
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	227 for key in range(maxkey + 1):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	228 if key not in table:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	229 mapvalue = None
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	230 mapcomment = 'UNDEFINED'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	231 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	232 mapvalue, mapcomment = table[key]
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	233 if mapvalue is None:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	234 mapchar = UNI_UNDEFINED
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	235 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	236 if isinstance(mapvalue, tuple):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	237 # 1-n mappings not supported
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	238 return None
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	239 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	240 mapchar = unichr(mapvalue)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	241 if mapcomment and comments:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	242 append(' %r\t# %s -> %s' % (mapchar,
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	243 hexrepr(key, key_precision),
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	244 mapcomment))
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	245 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	246 append(' %r' % mapchar)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	247
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	248 append(')')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	249 return l
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	250
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	251 def codegen(name, map, comments=1):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	252
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	253 """ Returns Python source for the given map.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	254
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	255 Comments are included in the source, if comments is true (default).
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	256
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	257 """
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	258 # Generate code
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	259 decoding_map_code = python_mapdef_code(
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	260 'decoding_map',
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	261 map,
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	262 comments=comments)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	263 decoding_table_code = python_tabledef_code(
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	264 'decoding_table',
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	265 map,
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	266 comments=comments)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	267 encoding_map_code = python_mapdef_code(
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	268 'encoding_map',
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	269 codecs.make_encoding_map(map),
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	270 comments=comments,
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	271 precisions=(4, 2))
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	272
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	273 l = [
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	274 '''\
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	275 """ Python Character Mapping Codec generated from '%s' with gencodec.py.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	276
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	277 """#"
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	278
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	279 import codecs
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	280
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	281 ### Codec APIs
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	282
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	283 class Codec(codecs.Codec):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	284
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	285 def encode(self,input,errors='strict'):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	286
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	287 return codecs.charmap_encode(input,errors,encoding_map)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	288
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	289 def decode(self,input,errors='strict'):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	290 ''' % name
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	291 ]
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	292 if decoding_table_code:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	293 l.append('''\
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	294 return codecs.charmap_decode(input,errors,decoding_table)''')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	295 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	296 l.append('''\
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	297 return codecs.charmap_decode(input,errors,decoding_map)''')
36121 686306a67e9b Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 35800 diff changeset	298
686306a67e9b Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 35800 diff changeset	299 l.append('''
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	300 class StreamWriter(Codec,codecs.StreamWriter):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	301 pass
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	302
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	303 class StreamReader(Codec,codecs.StreamReader):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	304 pass
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	305
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	306 ### encodings module API
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	307
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	308 def getregentry():
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	309
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	310 return (Codec().encode,Codec().decode,StreamReader,StreamWriter)
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	311 ''')
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	312
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	313 # Add decoding table or map (with preference to the table)
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	314 if not decoding_table_code:
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	315 l.append('''
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	316 ### Decoding Map
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	317 ''')
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	318 l.extend(decoding_map_code)
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	319 else:
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	320 l.append('''
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	321 ### Decoding Table
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	322 ''')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	323 l.extend(decoding_table_code)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	324
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	325 # Add encoding map
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	326 l.append('''
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	327 ### Encoding Map
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	328 ''')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	329 l.extend(encoding_map_code)
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	330
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	331 # Final new-line
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	332 l.append('\n')
36121 686306a67e9b Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 35800 diff changeset	333
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	334 return '\n'.join(l)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	335
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	336 def pymap(name,map,pyfile,comments=1):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	337
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	338 code = codegen(name,map,comments)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	339 f = open(pyfile,'w')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	340 f.write(code)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	341 f.close()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	342
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	343 def marshalmap(name,map,marshalfile):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	344
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	345 d = {}
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	346 for e,(u,c) in map.items():
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	347 d[e] = (u,c)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	348 f = open(marshalfile,'wb')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	349 marshal.dump(d,f)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	350 f.close()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	351
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	352 def convertdir(dir,prefix='',comments=1):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	353
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	354 mapnames = os.listdir(dir)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	355 for mapname in mapnames:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	356 mappathname = os.path.join(dir, mapname)
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	357 if not os.path.isfile(mappathname):
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	358 continue
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	359 name = os.path.split(mapname)[1]
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	360 name = name.replace('-','_')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	361 name = name.split('.')[0]
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	362 name = name.lower()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	363 codefile = name + '.py'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	364 marshalfile = name + '.mapping'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	365 print 'converting %s to %s and %s' % (mapname,
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	366 prefix + codefile,
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	367 prefix + marshalfile)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	368 try:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	369 map = readmap(os.path.join(dir,mapname))
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	370 if not map:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	371 print '* map is empty; skipping'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	372 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	373 pymap(mappathname, map, prefix + codefile,comments)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	374 marshalmap(mappathname, map, prefix + marshalfile)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	375 except ValueError, why:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	376 print '* conversion failed: %s' % why
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	377 raise
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	378
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	379 def rewritepythondir(dir,prefix='',comments=1):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	380
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	381 mapnames = os.listdir(dir)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	382 for mapname in mapnames:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	383 if not mapname.endswith('.mapping'):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	384 continue
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	385 codefile = mapname[:-len('.mapping')] + '.py'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	386 print 'converting %s to %s' % (mapname,
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	387 prefix + codefile)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	388 try:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	389 map = marshal.load(open(os.path.join(dir,mapname),
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	390 'rb'))
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	391 if not map:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	392 print '* map is empty; skipping'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	393 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	394 pymap(mapname, map, prefix + codefile,comments)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	395 except ValueError, why:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	396 print '* conversion failed: %s' % why
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	397
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	398 if __name__ == '__main__':
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	399
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	400 import sys
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	401 if 1:
37048 c048a09d06ff Remove apply() Neal Norwitz <nnorwitz@gmail.com> parents: 36121 diff changeset	402 convertdir(*sys.argv[1:])
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	403 else:
37048 c048a09d06ff Remove apply() Neal Norwitz <nnorwitz@gmail.com> parents: 36121 diff changeset	404 rewritepythondir(*sys.argv[1:])

Mercurial > cpython-lru_cache-weakref

annotate Tools/unicode/gencodec.py @ 37850:ebe59cd18273