cpython-lru_cache-weakref: Lib/test/test_multibytecodec

annotate Lib/test/test_multibytecodec_support.py @ 58925:eecf1bef052d 2.6

Merged revisions 78207 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r78207 | ezio.melotti | 2010-02-17 01:26:09 +0200 (Wed, 17 Feb 2010) | 1 line #7930: fix stripid ........

author	Ezio Melotti <ezio.melotti@gmail.com>
date	Tue, 16 Feb 2010 23:29:44 +0000
parents	b69017762208
children	86402fa0c5b8

rev	line source
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1 #!/usr/bin/env python
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	2 #
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	3 # test_multibytecodec_support.py
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	4 # Common Unittest Routines for CJK codecs
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	5 #
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	6
45468 1d84bf5f9478 Patch #2167 from calvin: Remove unused imports Christian Heimes <christian@cheimes.de> parents: 45235 diff changeset	7 import sys, codecs
42447 3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	8 import unittest, re
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	9 from test import test_support
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	10 from StringIO import StringIO
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	11
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	12 class TestBase:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	13 encoding = '' # codec name
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	14 codec = None # codec tuple (with 4 elements)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	15 tstring = '' # string to test StreamReader
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	16
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	17 codectests = None # must set. codec test tuple
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	18 roundtriptest = 1 # set if roundtrip is possible with unicode
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	19 has_iso10646 = 0 # set if this encoding contains whole iso10646 map
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	20 xmlcharnametest = None # string to test xmlcharrefreplace
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	21 unmappedunicode = u'\udeee' # a unicode codepoint that is not mapped.
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	22
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	23 def setUp(self):
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	24 if self.codec is None:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	25 self.codec = codecs.lookup(self.encoding)
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	26 self.encode = self.codec.encode
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	27 self.decode = self.codec.decode
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	28 self.reader = self.codec.streamreader
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	29 self.writer = self.codec.streamwriter
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	30 self.incrementalencoder = self.codec.incrementalencoder
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	31 self.incrementaldecoder = self.codec.incrementaldecoder
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	32
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	33 def test_chunkcoding(self):
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	34 for native, utf8 in zip(*[StringIO(f).readlines()
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	35 for f in self.tstring]):
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	36 u = self.decode(native)[0]
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	37 self.assertEqual(u, utf8.decode('utf-8'))
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	38 if self.roundtriptest:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	39 self.assertEqual(native, self.encode(u)[0])
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	40
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	41 def test_errorhandle(self):
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	42 for source, scheme, expected in self.codectests:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	43 if type(source) == type(''):
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	44 func = self.decode
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	45 else:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	46 func = self.encode
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	47 if expected:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	48 result = func(source, scheme)[0]
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	49 self.assertEqual(result, expected)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	50 else:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	51 self.assertRaises(UnicodeError, func, source, scheme)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	52
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	53 def test_xmlcharrefreplace(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	54 if self.has_iso10646:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	55 return
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	56
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	57 s = u"\u0b13\u0b23\u0b60 nd eggs"
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	58 self.assertEqual(
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	59 self.encode(s, "xmlcharrefreplace")[0],
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	60 "ଓଣୠ nd eggs"
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	61 )
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	62
37223 c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	63 def test_customreplace_encode(self):
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	64 if self.has_iso10646:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	65 return
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	66
47177 454d38c1f47e revert creation of the html.entities and html.parser modules Fred Drake <fdrake@acm.org> parents: 47143 diff changeset	67 from htmlentitydefs import codepoint2name
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	68
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	69 def xmlcharnamereplace(exc):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	70 if not isinstance(exc, UnicodeEncodeError):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	71 raise TypeError("don't know how to handle %r" % exc)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	72 l = []
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	73 for c in exc.object[exc.start:exc.end]:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	74 if ord(c) in codepoint2name:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	75 l.append(u"&%s;" % codepoint2name[ord(c)])
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	76 else:
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	77 l.append(u"&#%d;" % ord(c))
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	78 return (u"".join(l), exc.end)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	79
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	80 codecs.register_error("test.xmlcharnamereplace", xmlcharnamereplace)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	81
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	82 if self.xmlcharnametest:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	83 sin, sout = self.xmlcharnametest
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	84 else:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	85 sin = u"\xab\u211c\xbb = \u2329\u1234\u232a"
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	86 sout = "«&real;» = &lang;ሴ&rang;"
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	87 self.assertEqual(self.encode(sin,
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	88 "test.xmlcharnamereplace")[0], sout)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	89
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	90 def test_callback_wrong_objects(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	91 def myreplace(exc):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	92 return (ret, exc.end)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	93 codecs.register_error("test.cjktest", myreplace)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	94
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	95 for ret in ([1, 2, 3], [], None, object(), 'string', ''):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	96 self.assertRaises(TypeError, self.encode, self.unmappedunicode,
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	97 'test.cjktest')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	98
37223 c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	99 def test_callback_long_index(self):
c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	100 def myreplace(exc):
c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	101 return (u'x', long(exc.end))
c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	102 codecs.register_error("test.cjktest", myreplace)
c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	103 self.assertEqual(self.encode(u'abcd' + self.unmappedunicode + u'efgh',
c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	104 'test.cjktest'), ('abcdxefgh', 9))
c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	105
c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	106 def myreplace(exc):
c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	107 return (u'x', sys.maxint + 1)
c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	108 codecs.register_error("test.cjktest", myreplace)
c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	109 self.assertRaises(IndexError, self.encode, self.unmappedunicode,
c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	110 'test.cjktest')
c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	111
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	112 def test_callback_None_index(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	113 def myreplace(exc):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	114 return (u'x', None)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	115 codecs.register_error("test.cjktest", myreplace)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	116 self.assertRaises(TypeError, self.encode, self.unmappedunicode,
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	117 'test.cjktest')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	118
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	119 def test_callback_backward_index(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	120 def myreplace(exc):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	121 if myreplace.limit > 0:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	122 myreplace.limit -= 1
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	123 return (u'REPLACED', 0)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	124 else:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	125 return (u'TERMINAL', exc.end)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	126 myreplace.limit = 3
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	127 codecs.register_error("test.cjktest", myreplace)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	128 self.assertEqual(self.encode(u'abcd' + self.unmappedunicode + u'efgh',
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	129 'test.cjktest'),
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	130 ('abcdREPLACEDabcdREPLACEDabcdREPLACEDabcdTERMINALefgh', 9))
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	131
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	132 def test_callback_forward_index(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	133 def myreplace(exc):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	134 return (u'REPLACED', exc.end + 2)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	135 codecs.register_error("test.cjktest", myreplace)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	136 self.assertEqual(self.encode(u'abcd' + self.unmappedunicode + u'efgh',
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	137 'test.cjktest'), ('abcdREPLACEDgh', 9))
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	138
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	139 def test_callback_index_outofbound(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	140 def myreplace(exc):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	141 return (u'TERM', 100)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	142 codecs.register_error("test.cjktest", myreplace)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	143 self.assertRaises(IndexError, self.encode, self.unmappedunicode,
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	144 'test.cjktest')
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	145
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	146 def test_incrementalencoder(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	147 UTF8Reader = codecs.getreader('utf-8')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	148 for sizehint in [None] + range(1, 33) + \
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	149 [64, 128, 256, 512, 1024]:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	150 istream = UTF8Reader(StringIO(self.tstring[1]))
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	151 ostream = StringIO()
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	152 encoder = self.incrementalencoder()
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	153 while 1:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	154 if sizehint is not None:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	155 data = istream.read(sizehint)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	156 else:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	157 data = istream.read()
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	158
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	159 if not data:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	160 break
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	161 e = encoder.encode(data)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	162 ostream.write(e)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	163
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	164 self.assertEqual(ostream.getvalue(), self.tstring[0])
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	165
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	166 def test_incrementaldecoder(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	167 UTF8Writer = codecs.getwriter('utf-8')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	168 for sizehint in [None, -1] + range(1, 33) + \
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	169 [64, 128, 256, 512, 1024]:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	170 istream = StringIO(self.tstring[0])
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	171 ostream = UTF8Writer(StringIO())
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	172 decoder = self.incrementaldecoder()
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	173 while 1:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	174 data = istream.read(sizehint)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	175 if not data:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	176 break
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	177 else:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	178 u = decoder.decode(data)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	179 ostream.write(u)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	180
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	181 self.assertEqual(ostream.getvalue(), self.tstring[1])
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	182
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	183 def test_incrementalencoder_error_callback(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	184 inv = self.unmappedunicode
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	185
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	186 e = self.incrementalencoder()
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	187 self.assertRaises(UnicodeEncodeError, e.encode, inv, True)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	188
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	189 e.errors = 'ignore'
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	190 self.assertEqual(e.encode(inv, True), '')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	191
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	192 e.reset()
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	193 def tempreplace(exc):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	194 return (u'called', exc.end)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	195 codecs.register_error('test.incremental_error_callback', tempreplace)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	196 e.errors = 'test.incremental_error_callback'
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	197 self.assertEqual(e.encode(inv, True), 'called')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	198
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	199 # again
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	200 e.errors = 'ignore'
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	201 self.assertEqual(e.encode(inv, True), '')
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	202
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	203 def test_streamreader(self):
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	204 UTF8Writer = codecs.getwriter('utf-8')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	205 for name in ["read", "readline", "readlines"]:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	206 for sizehint in [None, -1] + range(1, 33) + \
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	207 [64, 128, 256, 512, 1024]:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	208 istream = self.reader(StringIO(self.tstring[0]))
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	209 ostream = UTF8Writer(StringIO())
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	210 func = getattr(istream, name)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	211 while 1:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	212 data = func(sizehint)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	213 if not data:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	214 break
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	215 if name == "readlines":
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	216 ostream.writelines(data)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	217 else:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	218 ostream.write(data)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	219
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	220 self.assertEqual(ostream.getvalue(), self.tstring[1])
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	221
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	222 def test_streamwriter(self):
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	223 readfuncs = ('read', 'readline', 'readlines')
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	224 UTF8Reader = codecs.getreader('utf-8')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	225 for name in readfuncs:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	226 for sizehint in [None] + range(1, 33) + \
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	227 [64, 128, 256, 512, 1024]:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	228 istream = UTF8Reader(StringIO(self.tstring[1]))
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	229 ostream = self.writer(StringIO())
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	230 func = getattr(istream, name)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	231 while 1:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	232 if sizehint is not None:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	233 data = func(sizehint)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	234 else:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	235 data = func()
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	236
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	237 if not data:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	238 break
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	239 if name == "readlines":
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	240 ostream.writelines(data)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	241 else:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	242 ostream.write(data)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	243
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	244 self.assertEqual(ostream.getvalue(), self.tstring[0])
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	245
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	246 if len(u'\U00012345') == 2: # ucs2 build
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	247 _unichr = unichr
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	248 def unichr(v):
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	249 if v >= 0x10000:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	250 return _unichr(0xd800 + ((v - 0x10000) >> 10)) + \
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	251 _unichr(0xdc00 + ((v - 0x10000) & 0x3ff))
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	252 else:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	253 return _unichr(v)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	254 _ord = ord
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	255 def ord(c):
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	256 if len(c) == 2:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	257 return 0x10000 + ((_ord(c[0]) - 0xd800) << 10) + \
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	258 (ord(c[1]) - 0xdc00)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	259 else:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	260 return _ord(c)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	261
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	262 class TestBase_Mapping(unittest.TestCase):
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	263 pass_enctest = []
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	264 pass_dectest = []
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	265 supmaps = []
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	266
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	267 def __init__(self, args, *kw):
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	268 unittest.TestCase.__init__(self, args, *kw)
57029 b69017762208 (neither to 2.6) Antoine Pitrou <solipsis@pitrou.net> parents: 47177 diff changeset	269 try:
b69017762208 (neither to 2.6) Antoine Pitrou <solipsis@pitrou.net> parents: 47177 diff changeset	270 self.open_mapping_file() # test it to report the error early
b69017762208 (neither to 2.6) Antoine Pitrou <solipsis@pitrou.net> parents: 47177 diff changeset	271 except IOError:
b69017762208 (neither to 2.6) Antoine Pitrou <solipsis@pitrou.net> parents: 47177 diff changeset	272 raise test_support.TestSkipped("Could not retrieve "+self.mapfileurl)
35998 96d1fa36964d Patch #1276356: Implement new resource "urlfetch" for regrtest. Hye-Shik Chang <hyeshik@gmail.com> parents: 32727 diff changeset	273
96d1fa36964d Patch #1276356: Implement new resource "urlfetch" for regrtest. Hye-Shik Chang <hyeshik@gmail.com> parents: 32727 diff changeset	274 def open_mapping_file(self):
96d1fa36964d Patch #1276356: Implement new resource "urlfetch" for regrtest. Hye-Shik Chang <hyeshik@gmail.com> parents: 32727 diff changeset	275 return test_support.open_urlresource(self.mapfileurl)
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	276
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	277 def test_mapping_file(self):
42447 3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	278 if self.mapfileurl.endswith('.xml'):
3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	279 self._test_mapping_file_ucm()
3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	280 else:
3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	281 self._test_mapping_file_plain()
3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	282
3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	283 def _test_mapping_file_plain(self):
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	284 unichrs = lambda s: u''.join(map(unichr, map(eval, s.split('+'))))
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	285 urt_wa = {}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	286
35998 96d1fa36964d Patch #1276356: Implement new resource "urlfetch" for regrtest. Hye-Shik Chang <hyeshik@gmail.com> parents: 32727 diff changeset	287 for line in self.open_mapping_file():
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	288 if not line:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	289 break
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	290 data = line.split('#')[0].strip().split()
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	291 if len(data) != 2:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	292 continue
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	293
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	294 csetval = eval(data[0])
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	295 if csetval <= 0x7F:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	296 csetch = chr(csetval & 0xff)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	297 elif csetval >= 0x1000000:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	298 csetch = chr(csetval >> 24) + chr((csetval >> 16) & 0xff) + \
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	299 chr((csetval >> 8) & 0xff) + chr(csetval & 0xff)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	300 elif csetval >= 0x10000:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	301 csetch = chr(csetval >> 16) + \
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	302 chr((csetval >> 8) & 0xff) + chr(csetval & 0xff)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	303 elif csetval >= 0x100:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	304 csetch = chr(csetval >> 8) + chr(csetval & 0xff)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	305 else:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	306 continue
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	307
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	308 unich = unichrs(data[1])
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	309 if ord(unich) == 0xfffd or urt_wa.has_key(unich):
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	310 continue
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	311 urt_wa[unich] = csetch
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	312
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	313 self._testpoint(csetch, unich)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	314
42447 3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	315 def _test_mapping_file_ucm(self):
3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	316 ucmdata = self.open_mapping_file().read()
3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	317 uc = re.findall('<a u="([A-F0-9]{4})" b="([0-9A-F ]+)"/>', ucmdata)
3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	318 for uni, coded in uc:
3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	319 unich = unichr(int(uni, 16))
3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	320 codech = ''.join(chr(int(c, 16)) for c in coded.split())
3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	321 self._testpoint(codech, unich)
3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	322
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	323 def test_mapping_supplemental(self):
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	324 for mapping in self.supmaps:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	325 self._testpoint(*mapping)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	326
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	327 def _testpoint(self, csetch, unich):
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	328 if (csetch, unich) not in self.pass_enctest:
45235 eca90b8aa7fd Update big5hkscs codec to conform to the HKSCS:2004 revision. Hye-Shik Chang <hyeshik@gmail.com> parents: 42447 diff changeset	329 try:
eca90b8aa7fd Update big5hkscs codec to conform to the HKSCS:2004 revision. Hye-Shik Chang <hyeshik@gmail.com> parents: 42447 diff changeset	330 self.assertEqual(unich.encode(self.encoding), csetch)
eca90b8aa7fd Update big5hkscs codec to conform to the HKSCS:2004 revision. Hye-Shik Chang <hyeshik@gmail.com> parents: 42447 diff changeset	331 except UnicodeError, exc:
eca90b8aa7fd Update big5hkscs codec to conform to the HKSCS:2004 revision. Hye-Shik Chang <hyeshik@gmail.com> parents: 42447 diff changeset	332 self.fail('Encoding failed while testing %s -> %s: %s' % (
eca90b8aa7fd Update big5hkscs codec to conform to the HKSCS:2004 revision. Hye-Shik Chang <hyeshik@gmail.com> parents: 42447 diff changeset	333 repr(unich), repr(csetch), exc.reason))
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	334 if (csetch, unich) not in self.pass_dectest:
45235 eca90b8aa7fd Update big5hkscs codec to conform to the HKSCS:2004 revision. Hye-Shik Chang <hyeshik@gmail.com> parents: 42447 diff changeset	335 try:
eca90b8aa7fd Update big5hkscs codec to conform to the HKSCS:2004 revision. Hye-Shik Chang <hyeshik@gmail.com> parents: 42447 diff changeset	336 self.assertEqual(csetch.decode(self.encoding), unich)
eca90b8aa7fd Update big5hkscs codec to conform to the HKSCS:2004 revision. Hye-Shik Chang <hyeshik@gmail.com> parents: 42447 diff changeset	337 except UnicodeError, exc:
eca90b8aa7fd Update big5hkscs codec to conform to the HKSCS:2004 revision. Hye-Shik Chang <hyeshik@gmail.com> parents: 42447 diff changeset	338 self.fail('Decoding failed while testing %s -> %s: %s' % (
eca90b8aa7fd Update big5hkscs codec to conform to the HKSCS:2004 revision. Hye-Shik Chang <hyeshik@gmail.com> parents: 42447 diff changeset	339 repr(csetch), repr(unich), exc.reason))
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	340
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	341 def load_teststring(encoding):
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	342 from test import cjkencodings_test
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	343 return cjkencodings_test.teststring[encoding]

Mercurial > cpython-lru_cache-weakref

annotate Lib/test/test_multibytecodec_support.py @ 58925:eecf1bef052d 2.6