cpython-lru_cache-weakref: Tools/unicode/gencodec.py annotate

annotate Tools/unicode/gencodec.py @ 80661:94de77bd0b4b 2.7

Issue #16194: document imp.load_dynamic problems when read several extension modules from the same binary object file.

author	Andrew Svetlov <andrew.svetlov@gmail.com>
date	Thu, 29 Nov 2012 19:51:28 +0200
parents	8aa5530e8407
children

rev	line source
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	1 """ Unicode Mapping Parser and Codec Generator.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	2
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	3 This script parses Unicode mapping files as available from the Unicode
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	4 site (ftp://ftp.unicode.org/Public/MAPPINGS/) and creates Python codec
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	5 modules from them. The codecs use the standard character mapping codec
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	6 to actually apply the mapping.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	7
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	8 Synopsis: gencodec.py dir codec_prefix
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	9
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	10 All files in dir are scanned and those producing non-empty mappings
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	11 will be written to <codec_prefix><mapname>.py with <mapname> being the
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	12 first part of the map's filename ('a' in a.b.c.txt) converted to
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	13 lowercase with hyphens replaced by underscores.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	14
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	15 The tool also writes marshalled versions of the mapping tables to the
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	16 same location (with .mapping extension).
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	17
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	18 Written by Marc-Andre Lemburg (mal@lemburg.com).
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	19
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	20 (c) Copyright CNRI, All Rights Reserved. NO WARRANTY.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	21 (c) Copyright Guido van Rossum, 2000.
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	22
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	23 Table generation:
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	24 (c) Copyright Marc-Andre Lemburg, 2005.
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	25 Licensed to PSF under a Contributor Agreement.
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	26
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	27 """#"
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	28
45468 1d84bf5f9478 Patch #2167 from calvin: Remove unused imports Christian Heimes <christian@cheimes.de> parents: 38586 diff changeset	29 import re, os, marshal, codecs
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	30
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	31 # Maximum allowed size of charmap tables
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	32 MAX_TABLE_SIZE = 8192
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	33
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	34 # Standard undefined Unicode code point
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	35 UNI_UNDEFINED = unichr(0xFFFE)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	36
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	37 mapRE = re.compile('((?:0x[0-9a-fA-F]+\+?)+)'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	38 '\s+'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	39 '((?:(?:0x[0-9a-fA-Z]+\|<[A-Za-z]+>)\+?)*)'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	40 '\s*'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	41 '(#.+)?')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	42
59640 8aa5530e8407 Remove py3k deprecation warnings from these Unicode tools. Florent Xicluna <florent.xicluna@gmail.com> parents: 45468 diff changeset	43 def parsecodes(codes, len=len, range=range):
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	44
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	45 """ Converts code combinations to either a single code integer
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	46 or a tuple of integers.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	47
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	48 meta-codes (in angular brackets, e.g. <LR> and <RL>) are
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	49 ignored.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	50
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	51 Empty codes or illegal ones are returned as None.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	52
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	53 """
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	54 if not codes:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	55 return None
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	56 l = codes.split('+')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	57 if len(l) == 1:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	58 return int(l[0],16)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	59 for i in range(len(l)):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	60 try:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	61 l[i] = int(l[i],16)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	62 except ValueError:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	63 l[i] = None
59640 8aa5530e8407 Remove py3k deprecation warnings from these Unicode tools. Florent Xicluna <florent.xicluna@gmail.com> parents: 45468 diff changeset	64 l = [x for x in l if x is not None]
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	65 if len(l) == 1:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	66 return l[0]
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	67 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	68 return tuple(l)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	69
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	70 def readmap(filename):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	71
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	72 f = open(filename,'r')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	73 lines = f.readlines()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	74 f.close()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	75 enc2uni = {}
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	76 identity = []
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	77 unmapped = range(256)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	78
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	79 # UTC mapping tables per convention don't include the identity
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	80 # mappings for code points 0x00 - 0x1F and 0x7F, unless these are
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	81 # explicitly mapped to different characters or undefined
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	82 for i in range(32) + [127]:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	83 identity.append(i)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	84 unmapped.remove(i)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	85 enc2uni[i] = (i, 'CONTROL CHARACTER')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	86
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	87 for line in lines:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	88 line = line.strip()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	89 if not line or line[0] == '#':
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	90 continue
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	91 m = mapRE.match(line)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	92 if not m:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	93 #print '* not matched: %s' % repr(line)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	94 continue
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	95 enc,uni,comment = m.groups()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	96 enc = parsecodes(enc)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	97 uni = parsecodes(uni)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	98 if comment is None:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	99 comment = ''
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	100 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	101 comment = comment[1:].strip()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	102 if enc < 256:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	103 if enc in unmapped:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	104 unmapped.remove(enc)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	105 if enc == uni:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	106 identity.append(enc)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	107 enc2uni[enc] = (uni,comment)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	108 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	109 enc2uni[enc] = (uni,comment)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	110
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	111 # If there are more identity-mapped entries than unmapped entries,
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	112 # it pays to generate an identity dictionary first, and add explicit
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	113 # mappings to None for the rest
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	114 if len(identity) >= len(unmapped):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	115 for enc in unmapped:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	116 enc2uni[enc] = (None, "")
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	117 enc2uni['IDENTITY'] = 256
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	118
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	119 return enc2uni
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	120
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	121 def hexrepr(t, precision=4):
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	122
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	123 if t is None:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	124 return 'None'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	125 try:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	126 len(t)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	127 except:
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	128 return '0x%0*X' % (precision, t)
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	129 try:
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	130 return '(' + ', '.join(['0x%0*X' % (precision, item)
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	131 for item in t]) + ')'
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	132 except TypeError, why:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	133 print '* failed to convert %r: %s' % (t, why)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	134 raise
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	135
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	136 def python_mapdef_code(varname, map, comments=1, precisions=(2, 4)):
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	137
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	138 l = []
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	139 append = l.append
59640 8aa5530e8407 Remove py3k deprecation warnings from these Unicode tools. Florent Xicluna <florent.xicluna@gmail.com> parents: 45468 diff changeset	140 if "IDENTITY" in map:
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	141 append("%s = codecs.make_identity_dict(range(%d))" %
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	142 (varname, map["IDENTITY"]))
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	143 append("%s.update({" % varname)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	144 splits = 1
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	145 del map["IDENTITY"]
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	146 identity = 1
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	147 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	148 append("%s = {" % varname)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	149 splits = 0
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	150 identity = 0
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	151
59640 8aa5530e8407 Remove py3k deprecation warnings from these Unicode tools. Florent Xicluna <florent.xicluna@gmail.com> parents: 45468 diff changeset	152 mappings = sorted(map.items())
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	153 i = 0
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	154 key_precision, value_precision = precisions
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	155 for mapkey, mapvalue in mappings:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	156 mapcomment = ''
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	157 if isinstance(mapkey, tuple):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	158 (mapkey, mapcomment) = mapkey
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	159 if isinstance(mapvalue, tuple):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	160 (mapvalue, mapcomment) = mapvalue
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	161 if mapkey is None:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	162 continue
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	163 if (identity and
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	164 mapkey == mapvalue and
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	165 mapkey < 256):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	166 # No need to include identity mappings, since these
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	167 # are already set for the first 256 code points.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	168 continue
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	169 key = hexrepr(mapkey, key_precision)
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	170 value = hexrepr(mapvalue, value_precision)
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	171 if mapcomment and comments:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	172 append(' %s: %s,\t# %s' % (key, value, mapcomment))
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	173 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	174 append(' %s: %s,' % (key, value))
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	175 i += 1
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	176 if i == 4096:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	177 # Split the definition into parts to that the Python
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	178 # parser doesn't dump core
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	179 if splits == 0:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	180 append('}')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	181 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	182 append('})')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	183 append('%s.update({' % varname)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	184 i = 0
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	185 splits = splits + 1
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	186 if splits == 0:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	187 append('}')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	188 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	189 append('})')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	190
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	191 return l
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	192
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	193 def python_tabledef_code(varname, map, comments=1, key_precision=2):
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	194
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	195 l = []
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	196 append = l.append
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	197 append('%s = (' % varname)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	198
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	199 # Analyze map and create table dict
59640 8aa5530e8407 Remove py3k deprecation warnings from these Unicode tools. Florent Xicluna <florent.xicluna@gmail.com> parents: 45468 diff changeset	200 mappings = sorted(map.items())
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	201 table = {}
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	202 maxkey = 0
59640 8aa5530e8407 Remove py3k deprecation warnings from these Unicode tools. Florent Xicluna <florent.xicluna@gmail.com> parents: 45468 diff changeset	203 if 'IDENTITY' in map:
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	204 for key in range(256):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	205 table[key] = (key, '')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	206 maxkey = 255
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	207 del map['IDENTITY']
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	208 for mapkey, mapvalue in mappings:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	209 mapcomment = ''
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	210 if isinstance(mapkey, tuple):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	211 (mapkey, mapcomment) = mapkey
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	212 if isinstance(mapvalue, tuple):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	213 (mapvalue, mapcomment) = mapvalue
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	214 if mapkey is None:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	215 continue
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	216 table[mapkey] = (mapvalue, mapcomment)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	217 if mapkey > maxkey:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	218 maxkey = mapkey
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	219 if maxkey > MAX_TABLE_SIZE:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	220 # Table too large
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	221 return None
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	222
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	223 # Create table code
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	224 for key in range(maxkey + 1):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	225 if key not in table:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	226 mapvalue = None
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	227 mapcomment = 'UNDEFINED'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	228 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	229 mapvalue, mapcomment = table[key]
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	230 if mapvalue is None:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	231 mapchar = UNI_UNDEFINED
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	232 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	233 if isinstance(mapvalue, tuple):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	234 # 1-n mappings not supported
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	235 return None
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	236 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	237 mapchar = unichr(mapvalue)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	238 if mapcomment and comments:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	239 append(' %r\t# %s -> %s' % (mapchar,
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	240 hexrepr(key, key_precision),
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	241 mapcomment))
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	242 else:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	243 append(' %r' % mapchar)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	244
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	245 append(')')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	246 return l
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	247
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	248 def codegen(name, map, encodingname, comments=1):
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	249
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	250 """ Returns Python source for the given map.
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	251
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	252 Comments are included in the source, if comments is true (default).
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	253
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	254 """
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	255 # Generate code
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	256 decoding_map_code = python_mapdef_code(
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	257 'decoding_map',
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	258 map,
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	259 comments=comments)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	260 decoding_table_code = python_tabledef_code(
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	261 'decoding_table',
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	262 map,
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	263 comments=comments)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	264 encoding_map_code = python_mapdef_code(
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	265 'encoding_map',
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	266 codecs.make_encoding_map(map),
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	267 comments=comments,
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	268 precisions=(4, 2))
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	269
38586 9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	270 if decoding_table_code:
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	271 suffix = 'table'
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	272 else:
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	273 suffix = 'map'
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	274
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	275 l = [
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	276 '''\
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	277 """ Python Character Mapping Codec %s generated from '%s' with gencodec.py.
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	278
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	279 """#"
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	280
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	281 import codecs
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	282
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	283 ### Codec APIs
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	284
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	285 class Codec(codecs.Codec):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	286
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	287 def encode(self,input,errors='strict'):
38586 9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	288 return codecs.charmap_encode(input,errors,encoding_%s)
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	289
38586 9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	290 def decode(self,input,errors='strict'):
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	291 return codecs.charmap_decode(input,errors,decoding_%s)
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	292 ''' % (encodingname, name, suffix, suffix)]
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	293 l.append('''\
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	294 class IncrementalEncoder(codecs.IncrementalEncoder):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	295 def encode(self, input, final=False):
38586 9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	296 return codecs.charmap_encode(input,self.errors,encoding_%s)[0]
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	297
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	298 class IncrementalDecoder(codecs.IncrementalDecoder):
38586 9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	299 def decode(self, input, final=False):
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	300 return codecs.charmap_decode(input,self.errors,decoding_%s)[0]''' %
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	301 (suffix, suffix))
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	302
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	303 l.append('''
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	304 class StreamWriter(Codec,codecs.StreamWriter):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	305 pass
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	306
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	307 class StreamReader(Codec,codecs.StreamReader):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	308 pass
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	309
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	310 ### encodings module API
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	311
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	312 def getregentry():
38346 92b0fc54d230 when generating python code prefer to generate valid python code Jack Diederich <jackdied@gmail.com> parents: 37323 diff changeset	313 return codecs.CodecInfo(
92b0fc54d230 when generating python code prefer to generate valid python code Jack Diederich <jackdied@gmail.com> parents: 37323 diff changeset	314 name=%r,
38586 9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	315 encode=Codec().encode,
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	316 decode=Codec().decode,
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	317 incrementalencoder=IncrementalEncoder,
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	318 incrementaldecoder=IncrementalDecoder,
38586 9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	319 streamreader=StreamReader,
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	320 streamwriter=StreamWriter,
38346 92b0fc54d230 when generating python code prefer to generate valid python code Jack Diederich <jackdied@gmail.com> parents: 37323 diff changeset	321 )
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	322 ''' % encodingname.replace('_', '-'))
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	323
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	324 # Add decoding table or map (with preference to the table)
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	325 if not decoding_table_code:
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	326 l.append('''
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	327 ### Decoding Map
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	328 ''')
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	329 l.extend(decoding_map_code)
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	330 else:
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	331 l.append('''
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	332 ### Decoding Table
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	333 ''')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	334 l.extend(decoding_table_code)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	335
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	336 # Add encoding map
38586 9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	337 if decoding_table_code:
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	338 l.append('''
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	339 ### Encoding table
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	340 encoding_table=codecs.charmap_build(decoding_table)
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	341 ''')
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	342 else:
9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	343 l.append('''
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	344 ### Encoding Map
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	345 ''')
38586 9e0b050ce46f Patch #1359618: Speed-up charmap encoder. Martin v. Löwis <martin@v.loewis.de> parents: 38346 diff changeset	346 l.extend(encoding_map_code)
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	347
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	348 # Final new-line
37323 164f95e8e0cc Don't add multiple empty lines at the end of the codec. With this a Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	349 l.append('')
36121 686306a67e9b Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 35800 diff changeset	350
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	351 return '\n'.join(l).expandtabs()
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	352
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	353 def pymap(name,map,pyfile,encodingname,comments=1):
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	354
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	355 code = codegen(name,map,encodingname,comments)
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	356 f = open(pyfile,'w')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	357 f.write(code)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	358 f.close()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	359
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	360 def marshalmap(name,map,marshalfile):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	361
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	362 d = {}
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	363 for e,(u,c) in map.items():
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	364 d[e] = (u,c)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	365 f = open(marshalfile,'wb')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	366 marshal.dump(d,f)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	367 f.close()
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	368
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	369 def convertdir(dir, dirprefix='', nameprefix='', comments=1):
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	370
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	371 mapnames = os.listdir(dir)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	372 for mapname in mapnames:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	373 mappathname = os.path.join(dir, mapname)
35800 4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	374 if not os.path.isfile(mappathname):
4bf6459c8e3b Apply some cosmetic fixes to the output of the script. Marc-André Lemburg <mal@egenix.com> parents: 35760 diff changeset	375 continue
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	376 name = os.path.split(mapname)[1]
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	377 name = name.replace('-','_')
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	378 name = name.split('.')[0]
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	379 name = name.lower()
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	380 name = nameprefix + name
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	381 codefile = name + '.py'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	382 marshalfile = name + '.mapping'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	383 print 'converting %s to %s and %s' % (mapname,
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	384 dirprefix + codefile,
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	385 dirprefix + marshalfile)
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	386 try:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	387 map = readmap(os.path.join(dir,mapname))
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	388 if not map:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	389 print '* map is empty; skipping'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	390 else:
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	391 pymap(mappathname, map, dirprefix + codefile,name,comments)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	392 marshalmap(mappathname, map, dirprefix + marshalfile)
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	393 except ValueError, why:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	394 print '* conversion failed: %s' % why
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	395 raise
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	396
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	397 def rewritepythondir(dir, dirprefix='', comments=1):
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	398
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	399 mapnames = os.listdir(dir)
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	400 for mapname in mapnames:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	401 if not mapname.endswith('.mapping'):
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	402 continue
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	403 name = mapname[:-len('.mapping')]
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	404 codefile = name + '.py'
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	405 print 'converting %s to %s' % (mapname,
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	406 dirprefix + codefile)
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	407 try:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	408 map = marshal.load(open(os.path.join(dir,mapname),
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	409 'rb'))
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	410 if not map:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	411 print '* map is empty; skipping'
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	412 else:
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	413 pymap(mapname, map, dirprefix + codefile,name,comments)
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	414 except ValueError, why:
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	415 print '* conversion failed: %s' % why
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	416
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	417 if __name__ == '__main__':
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	418
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	419 import sys
79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	420 if 1:
59640 8aa5530e8407 Remove py3k deprecation warnings from these Unicode tools. Florent Xicluna <florent.xicluna@gmail.com> parents: 45468 diff changeset	421 convertdir(*sys.argv[1:])
35760 79c5b817fedc Moved gencodec.py to the Tools/unicode/ directory. Marc-André Lemburg <mal@egenix.com> parents: diff changeset	422 else:
59640 8aa5530e8407 Remove py3k deprecation warnings from these Unicode tools. Florent Xicluna <florent.xicluna@gmail.com> parents: 45468 diff changeset	423 rewritepythondir(*sys.argv[1:])

Mercurial > cpython-lru_cache-weakref

annotate Tools/unicode/gencodec.py @ 80661:94de77bd0b4b 2.7