cpython-lru_cache-weakref: Lib/test/test_multibytecodec

annotate Lib/test/test_multibytecodec_support.py @ 79675:c0a2294de787 2.7

Issue #16199: Backport 2to3 typo fix from 3.2.

author	Chris Jerdonek <chris.jerdonek@gmail.com>
date	Thu, 11 Oct 2012 16:15:46 -0700
parents	53912b58eee6
children

rev	line source
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	1 #!/usr/bin/env python
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	2 #
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	3 # test_multibytecodec_support.py
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	4 # Common Unittest Routines for CJK codecs
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	5 #
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	6
70178 83f4c270b27d Issue #12057: Fix .hgeol and test_multibytecodec_support for the conversion of Victor Stinner <victor.stinner@haypocalc.com> parents: 68256 diff changeset	7 import codecs
83f4c270b27d Issue #12057: Fix .hgeol and test_multibytecodec_support for the conversion of Victor Stinner <victor.stinner@haypocalc.com> parents: 68256 diff changeset	8 import os
83f4c270b27d Issue #12057: Fix .hgeol and test_multibytecodec_support for the conversion of Victor Stinner <victor.stinner@haypocalc.com> parents: 68256 diff changeset	9 import re
83f4c270b27d Issue #12057: Fix .hgeol and test_multibytecodec_support for the conversion of Victor Stinner <victor.stinner@haypocalc.com> parents: 68256 diff changeset	10 import sys
83f4c270b27d Issue #12057: Fix .hgeol and test_multibytecodec_support for the conversion of Victor Stinner <victor.stinner@haypocalc.com> parents: 68256 diff changeset	11 import unittest
57466 7b74abb80356 When open_urlresource() fails, HTTPException is another possible error Antoine Pitrou <solipsis@pitrou.net> parents: 57027 diff changeset	12 from httplib import HTTPException
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	13 from test import test_support
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	14 from StringIO import StringIO
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	15
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	16 class TestBase:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	17 encoding = '' # codec name
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	18 codec = None # codec tuple (with 4 elements)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	19 tstring = '' # string to test StreamReader
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	20
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	21 codectests = None # must set. codec test tuple
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	22 roundtriptest = 1 # set if roundtrip is possible with unicode
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	23 has_iso10646 = 0 # set if this encoding contains whole iso10646 map
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	24 xmlcharnametest = None # string to test xmlcharrefreplace
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	25 unmappedunicode = u'\udeee' # a unicode codepoint that is not mapped.
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	26
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	27 def setUp(self):
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	28 if self.codec is None:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	29 self.codec = codecs.lookup(self.encoding)
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	30 self.encode = self.codec.encode
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	31 self.decode = self.codec.decode
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	32 self.reader = self.codec.streamreader
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	33 self.writer = self.codec.streamwriter
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	34 self.incrementalencoder = self.codec.incrementalencoder
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	35 self.incrementaldecoder = self.codec.incrementaldecoder
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	36
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	37 def test_chunkcoding(self):
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	38 for native, utf8 in zip(*[StringIO(f).readlines()
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	39 for f in self.tstring]):
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	40 u = self.decode(native)[0]
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	41 self.assertEqual(u, utf8.decode('utf-8'))
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	42 if self.roundtriptest:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	43 self.assertEqual(native, self.encode(u)[0])
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	44
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	45 def test_errorhandle(self):
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	46 for source, scheme, expected in self.codectests:
70630 53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	47 if isinstance(source, bytes):
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	48 func = self.decode
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	49 else:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	50 func = self.encode
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	51 if expected:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	52 result = func(source, scheme)[0]
70630 53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	53 if func is self.decode:
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	54 self.assertTrue(type(result) is unicode, type(result))
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	55 self.assertEqual(result, expected,
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	56 '%r.decode(%r, %r)=%r != %r'
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	57 % (source, self.encoding, scheme, result,
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	58 expected))
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	59 else:
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	60 self.assertTrue(type(result) is bytes, type(result))
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	61 self.assertEqual(result, expected,
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	62 '%r.encode(%r, %r)=%r != %r'
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	63 % (source, self.encoding, scheme, result,
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	64 expected))
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	65 else:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	66 self.assertRaises(UnicodeError, func, source, scheme)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	67
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	68 def test_xmlcharrefreplace(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	69 if self.has_iso10646:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	70 return
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	71
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	72 s = u"\u0b13\u0b23\u0b60 nd eggs"
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	73 self.assertEqual(
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	74 self.encode(s, "xmlcharrefreplace")[0],
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	75 "ଓଣୠ nd eggs"
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	76 )
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	77
37223 c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	78 def test_customreplace_encode(self):
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	79 if self.has_iso10646:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	80 return
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	81
47177 454d38c1f47e revert creation of the html.entities and html.parser modules Fred Drake <fdrake@acm.org> parents: 47143 diff changeset	82 from htmlentitydefs import codepoint2name
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	83
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	84 def xmlcharnamereplace(exc):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	85 if not isinstance(exc, UnicodeEncodeError):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	86 raise TypeError("don't know how to handle %r" % exc)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	87 l = []
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	88 for c in exc.object[exc.start:exc.end]:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	89 if ord(c) in codepoint2name:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	90 l.append(u"&%s;" % codepoint2name[ord(c)])
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	91 else:
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	92 l.append(u"&#%d;" % ord(c))
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	93 return (u"".join(l), exc.end)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	94
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	95 codecs.register_error("test.xmlcharnamereplace", xmlcharnamereplace)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	96
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	97 if self.xmlcharnametest:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	98 sin, sout = self.xmlcharnametest
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	99 else:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	100 sin = u"\xab\u211c\xbb = \u2329\u1234\u232a"
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	101 sout = "«&real;» = &lang;ሴ&rang;"
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	102 self.assertEqual(self.encode(sin,
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	103 "test.xmlcharnamereplace")[0], sout)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	104
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	105 def test_callback_wrong_objects(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	106 def myreplace(exc):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	107 return (ret, exc.end)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	108 codecs.register_error("test.cjktest", myreplace)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	109
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	110 for ret in ([1, 2, 3], [], None, object(), 'string', ''):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	111 self.assertRaises(TypeError, self.encode, self.unmappedunicode,
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	112 'test.cjktest')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	113
37223 c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	114 def test_callback_long_index(self):
c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	115 def myreplace(exc):
c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	116 return (u'x', long(exc.end))
c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	117 codecs.register_error("test.cjktest", myreplace)
c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	118 self.assertEqual(self.encode(u'abcd' + self.unmappedunicode + u'efgh',
c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	119 'test.cjktest'), ('abcdxefgh', 9))
c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	120
c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	121 def myreplace(exc):
c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	122 return (u'x', sys.maxint + 1)
c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	123 codecs.register_error("test.cjktest", myreplace)
c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	124 self.assertRaises(IndexError, self.encode, self.unmappedunicode,
c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	125 'test.cjktest')
c0818f21f65a Allow long objects as a position value of error callbacks returned. Hye-Shik Chang <hyeshik@gmail.com> parents: 37218 diff changeset	126
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	127 def test_callback_None_index(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	128 def myreplace(exc):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	129 return (u'x', None)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	130 codecs.register_error("test.cjktest", myreplace)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	131 self.assertRaises(TypeError, self.encode, self.unmappedunicode,
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	132 'test.cjktest')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	133
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	134 def test_callback_backward_index(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	135 def myreplace(exc):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	136 if myreplace.limit > 0:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	137 myreplace.limit -= 1
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	138 return (u'REPLACED', 0)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	139 else:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	140 return (u'TERMINAL', exc.end)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	141 myreplace.limit = 3
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	142 codecs.register_error("test.cjktest", myreplace)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	143 self.assertEqual(self.encode(u'abcd' + self.unmappedunicode + u'efgh',
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	144 'test.cjktest'),
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	145 ('abcdREPLACEDabcdREPLACEDabcdREPLACEDabcdTERMINALefgh', 9))
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	146
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	147 def test_callback_forward_index(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	148 def myreplace(exc):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	149 return (u'REPLACED', exc.end + 2)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	150 codecs.register_error("test.cjktest", myreplace)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	151 self.assertEqual(self.encode(u'abcd' + self.unmappedunicode + u'efgh',
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	152 'test.cjktest'), ('abcdREPLACEDgh', 9))
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	153
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	154 def test_callback_index_outofbound(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	155 def myreplace(exc):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	156 return (u'TERM', 100)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	157 codecs.register_error("test.cjktest", myreplace)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	158 self.assertRaises(IndexError, self.encode, self.unmappedunicode,
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	159 'test.cjktest')
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	160
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	161 def test_incrementalencoder(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	162 UTF8Reader = codecs.getreader('utf-8')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	163 for sizehint in [None] + range(1, 33) + \
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	164 [64, 128, 256, 512, 1024]:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	165 istream = UTF8Reader(StringIO(self.tstring[1]))
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	166 ostream = StringIO()
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	167 encoder = self.incrementalencoder()
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	168 while 1:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	169 if sizehint is not None:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	170 data = istream.read(sizehint)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	171 else:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	172 data = istream.read()
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	173
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	174 if not data:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	175 break
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	176 e = encoder.encode(data)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	177 ostream.write(e)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	178
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	179 self.assertEqual(ostream.getvalue(), self.tstring[0])
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	180
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	181 def test_incrementaldecoder(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	182 UTF8Writer = codecs.getwriter('utf-8')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	183 for sizehint in [None, -1] + range(1, 33) + \
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	184 [64, 128, 256, 512, 1024]:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	185 istream = StringIO(self.tstring[0])
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	186 ostream = UTF8Writer(StringIO())
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	187 decoder = self.incrementaldecoder()
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	188 while 1:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	189 data = istream.read(sizehint)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	190 if not data:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	191 break
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	192 else:
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	193 u = decoder.decode(data)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	194 ostream.write(u)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	195
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	196 self.assertEqual(ostream.getvalue(), self.tstring[1])
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	197
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	198 def test_incrementalencoder_error_callback(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	199 inv = self.unmappedunicode
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	200
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	201 e = self.incrementalencoder()
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	202 self.assertRaises(UnicodeEncodeError, e.encode, inv, True)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	203
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	204 e.errors = 'ignore'
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	205 self.assertEqual(e.encode(inv, True), '')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	206
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	207 e.reset()
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	208 def tempreplace(exc):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	209 return (u'called', exc.end)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	210 codecs.register_error('test.incremental_error_callback', tempreplace)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	211 e.errors = 'test.incremental_error_callback'
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	212 self.assertEqual(e.encode(inv, True), 'called')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	213
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	214 # again
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	215 e.errors = 'ignore'
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	216 self.assertEqual(e.encode(inv, True), '')
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	217
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	218 def test_streamreader(self):
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	219 UTF8Writer = codecs.getwriter('utf-8')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	220 for name in ["read", "readline", "readlines"]:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	221 for sizehint in [None, -1] + range(1, 33) + \
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	222 [64, 128, 256, 512, 1024]:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	223 istream = self.reader(StringIO(self.tstring[0]))
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	224 ostream = UTF8Writer(StringIO())
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	225 func = getattr(istream, name)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	226 while 1:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	227 data = func(sizehint)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	228 if not data:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	229 break
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	230 if name == "readlines":
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	231 ostream.writelines(data)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	232 else:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	233 ostream.write(data)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	234
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	235 self.assertEqual(ostream.getvalue(), self.tstring[1])
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	236
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	237 def test_streamwriter(self):
37218 e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs. Hye-Shik Chang <hyeshik@gmail.com> parents: 35998 diff changeset	238 readfuncs = ('read', 'readline', 'readlines')
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	239 UTF8Reader = codecs.getreader('utf-8')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	240 for name in readfuncs:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	241 for sizehint in [None] + range(1, 33) + \
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	242 [64, 128, 256, 512, 1024]:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	243 istream = UTF8Reader(StringIO(self.tstring[1]))
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	244 ostream = self.writer(StringIO())
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	245 func = getattr(istream, name)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	246 while 1:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	247 if sizehint is not None:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	248 data = func(sizehint)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	249 else:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	250 data = func()
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	251
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	252 if not data:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	253 break
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	254 if name == "readlines":
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	255 ostream.writelines(data)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	256 else:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	257 ostream.write(data)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	258
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	259 self.assertEqual(ostream.getvalue(), self.tstring[0])
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	260
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	261 class TestBase_Mapping(unittest.TestCase):
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	262 pass_enctest = []
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	263 pass_dectest = []
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	264 supmaps = []
70630 53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	265 codectests = []
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	266
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	267 def __init__(self, args, *kw):
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	268 unittest.TestCase.__init__(self, args, *kw)
57029 b69017762208 (neither to 2.6) Antoine Pitrou <solipsis@pitrou.net> parents: 47177 diff changeset	269 try:
65339 fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	270 self.open_mapping_file().close() # test it to report the error early
57466 7b74abb80356 When open_urlresource() fails, HTTPException is another possible error Antoine Pitrou <solipsis@pitrou.net> parents: 57027 diff changeset	271 except (IOError, HTTPException):
57027 ff8e359e2808 Hum, test skipping when the URL isn't reachable hadn't been applied to trunk. Antoine Pitrou <solipsis@pitrou.net> parents: 47177 diff changeset	272 self.skipTest("Could not retrieve "+self.mapfileurl)
35998 96d1fa36964d Patch #1276356: Implement new resource "urlfetch" for regrtest. Hye-Shik Chang <hyeshik@gmail.com> parents: 32727 diff changeset	273
96d1fa36964d Patch #1276356: Implement new resource "urlfetch" for regrtest. Hye-Shik Chang <hyeshik@gmail.com> parents: 32727 diff changeset	274 def open_mapping_file(self):
96d1fa36964d Patch #1276356: Implement new resource "urlfetch" for regrtest. Hye-Shik Chang <hyeshik@gmail.com> parents: 32727 diff changeset	275 return test_support.open_urlresource(self.mapfileurl)
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	276
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	277 def test_mapping_file(self):
42447 3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	278 if self.mapfileurl.endswith('.xml'):
3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	279 self._test_mapping_file_ucm()
3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	280 else:
3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	281 self._test_mapping_file_plain()
3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	282
3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	283 def _test_mapping_file_plain(self):
59184 86402fa0c5b8 Merged revisions 78497 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 57029 diff changeset	284 _unichr = lambda c: eval("u'\\U%08x'" % int(c, 16))
86402fa0c5b8 Merged revisions 78497 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 57029 diff changeset	285 unichrs = lambda s: u''.join(_unichr(c) for c in s.split('+'))
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	286 urt_wa = {}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	287
65339 fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	288 with self.open_mapping_file() as f:
fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	289 for line in f:
fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	290 if not line:
fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	291 break
fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	292 data = line.split('#')[0].strip().split()
fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	293 if len(data) != 2:
fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	294 continue
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	295
65339 fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	296 csetval = eval(data[0])
fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	297 if csetval <= 0x7F:
fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	298 csetch = chr(csetval & 0xff)
fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	299 elif csetval >= 0x1000000:
fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	300 csetch = chr(csetval >> 24) + chr((csetval >> 16) & 0xff) + \
fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	301 chr((csetval >> 8) & 0xff) + chr(csetval & 0xff)
fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	302 elif csetval >= 0x10000:
fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	303 csetch = chr(csetval >> 16) + \
fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	304 chr((csetval >> 8) & 0xff) + chr(csetval & 0xff)
fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	305 elif csetval >= 0x100:
fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	306 csetch = chr(csetval >> 8) + chr(csetval & 0xff)
fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	307 else:
fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	308 continue
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	309
65339 fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	310 unich = unichrs(data[1])
fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	311 if unich == u'\ufffd' or unich in urt_wa:
fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	312 continue
fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	313 urt_wa[unich] = csetch
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	314
65339 fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	315 self._testpoint(csetch, unich)
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	316
42447 3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	317 def _test_mapping_file_ucm(self):
65339 fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	318 with self.open_mapping_file() as f:
fea8f21b199d Merged revisions 85503 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 59183 diff changeset	319 ucmdata = f.read()
42447 3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	320 uc = re.findall('<a u="([A-F0-9]{4})" b="([0-9A-F ]+)"/>', ucmdata)
3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	321 for uni, coded in uc:
3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	322 unich = unichr(int(uni, 16))
3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	323 codech = ''.join(chr(int(c, 16)) for c in coded.split())
3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	324 self._testpoint(codech, unich)
3fe57daaa608 Fix gb18030 codec's bug that doesn't map two-byte characters on Hye-Shik Chang <hyeshik@gmail.com> parents: 37223 diff changeset	325
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	326 def test_mapping_supplemental(self):
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	327 for mapping in self.supmaps:
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	328 self._testpoint(*mapping)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	329
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	330 def _testpoint(self, csetch, unich):
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	331 if (csetch, unich) not in self.pass_enctest:
45235 eca90b8aa7fd Update big5hkscs codec to conform to the HKSCS:2004 revision. Hye-Shik Chang <hyeshik@gmail.com> parents: 42447 diff changeset	332 try:
eca90b8aa7fd Update big5hkscs codec to conform to the HKSCS:2004 revision. Hye-Shik Chang <hyeshik@gmail.com> parents: 42447 diff changeset	333 self.assertEqual(unich.encode(self.encoding), csetch)
eca90b8aa7fd Update big5hkscs codec to conform to the HKSCS:2004 revision. Hye-Shik Chang <hyeshik@gmail.com> parents: 42447 diff changeset	334 except UnicodeError, exc:
eca90b8aa7fd Update big5hkscs codec to conform to the HKSCS:2004 revision. Hye-Shik Chang <hyeshik@gmail.com> parents: 42447 diff changeset	335 self.fail('Encoding failed while testing %s -> %s: %s' % (
eca90b8aa7fd Update big5hkscs codec to conform to the HKSCS:2004 revision. Hye-Shik Chang <hyeshik@gmail.com> parents: 42447 diff changeset	336 repr(unich), repr(csetch), exc.reason))
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	337 if (csetch, unich) not in self.pass_dectest:
45235 eca90b8aa7fd Update big5hkscs codec to conform to the HKSCS:2004 revision. Hye-Shik Chang <hyeshik@gmail.com> parents: 42447 diff changeset	338 try:
eca90b8aa7fd Update big5hkscs codec to conform to the HKSCS:2004 revision. Hye-Shik Chang <hyeshik@gmail.com> parents: 42447 diff changeset	339 self.assertEqual(csetch.decode(self.encoding), unich)
eca90b8aa7fd Update big5hkscs codec to conform to the HKSCS:2004 revision. Hye-Shik Chang <hyeshik@gmail.com> parents: 42447 diff changeset	340 except UnicodeError, exc:
eca90b8aa7fd Update big5hkscs codec to conform to the HKSCS:2004 revision. Hye-Shik Chang <hyeshik@gmail.com> parents: 42447 diff changeset	341 self.fail('Decoding failed while testing %s -> %s: %s' % (
eca90b8aa7fd Update big5hkscs codec to conform to the HKSCS:2004 revision. Hye-Shik Chang <hyeshik@gmail.com> parents: 42447 diff changeset	342 repr(csetch), repr(unich), exc.reason))
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: diff changeset	343
70630 53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	344 def test_errorhandle(self):
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	345 for source, scheme, expected in self.codectests:
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	346 if isinstance(source, bytes):
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	347 func = source.decode
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	348 else:
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	349 func = source.encode
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	350 if expected:
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	351 if isinstance(source, bytes):
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	352 result = func(self.encoding, scheme)
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	353 self.assertTrue(type(result) is unicode, type(result))
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	354 self.assertEqual(result, expected,
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	355 '%r.decode(%r, %r)=%r != %r'
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	356 % (source, self.encoding, scheme, result,
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	357 expected))
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	358 else:
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	359 result = func(self.encoding, scheme)
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	360 self.assertTrue(type(result) is bytes, type(result))
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	361 self.assertEqual(result, expected,
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	362 '%r.encode(%r, %r)=%r != %r'
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	363 % (source, self.encoding, scheme, result,
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	364 expected))
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	365 else:
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	366 self.assertRaises(UnicodeError, func, self.encoding, scheme)
53912b58eee6 Issue #12016: Add test_errorhandle() to TestBase_Mapping of Victor Stinner <victor.stinner@haypocalc.com> parents: 70540 diff changeset	367
70178 83f4c270b27d Issue #12057: Fix .hgeol and test_multibytecodec_support for the conversion of Victor Stinner <victor.stinner@haypocalc.com> parents: 68256 diff changeset	368 def load_teststring(name):
83f4c270b27d Issue #12057: Fix .hgeol and test_multibytecodec_support for the conversion of Victor Stinner <victor.stinner@haypocalc.com> parents: 68256 diff changeset	369 dir = os.path.join(os.path.dirname(__file__), 'cjkencodings')
83f4c270b27d Issue #12057: Fix .hgeol and test_multibytecodec_support for the conversion of Victor Stinner <victor.stinner@haypocalc.com> parents: 68256 diff changeset	370 with open(os.path.join(dir, name + '.txt'), 'rb') as f:
83f4c270b27d Issue #12057: Fix .hgeol and test_multibytecodec_support for the conversion of Victor Stinner <victor.stinner@haypocalc.com> parents: 68256 diff changeset	371 encoded = f.read()
83f4c270b27d Issue #12057: Fix .hgeol and test_multibytecodec_support for the conversion of Victor Stinner <victor.stinner@haypocalc.com> parents: 68256 diff changeset	372 with open(os.path.join(dir, name + '-utf8.txt'), 'rb') as f:
83f4c270b27d Issue #12057: Fix .hgeol and test_multibytecodec_support for the conversion of Victor Stinner <victor.stinner@haypocalc.com> parents: 68256 diff changeset	373 utf8 = f.read()
83f4c270b27d Issue #12057: Fix .hgeol and test_multibytecodec_support for the conversion of Victor Stinner <victor.stinner@haypocalc.com> parents: 68256 diff changeset	374 return encoded, utf8

Mercurial > cpython-lru_cache-weakref

annotate Lib/test/test_multibytecodec_support.py @ 79675:c0a2294de787 2.7