Mercurial > cpython-lru_cache-weakref

annotate Lib/test/test_multibytecodec.py @ 58925:eecf1bef052d 2.6

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Merged revisions 78207 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r78207 | ezio.melotti | 2010-02-17 01:26:09 +0200 (Wed, 17 Feb 2010) | 1 line #7930: fix stripid ........
author Ezio Melotti <ezio.melotti@gmail.com>
date Tue, 16 Feb 2010 23:29:44 +0000
parents efe706aab32b
children 14d23bddf793 6768b2350c57
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
31386
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
1 #!/usr/bin/env python
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
2 #
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
3 # test_multibytecodec.py
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
4 # Unit test for multibytecodec itself
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
5 #
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
6
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
7 from test import test_support
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
8 from test import test_multibytecodec_support
39029
f93dbfcc663b Add a testcase for r47086 which fixed a bug in codec_getstreamcodec().
Hye-Shik Chang <hyeshik@gmail.com>
parents: 37925
diff changeset
9 from test.test_support import TESTFN
f93dbfcc663b Add a testcase for r47086 which fixed a bug in codec_getstreamcodec().
Hye-Shik Chang <hyeshik@gmail.com>
parents: 37925
diff changeset
10 import unittest, StringIO, codecs, sys, os
47989
efe706aab32b #3305: self->stream can be NULL.
Georg Brandl <georg@python.org>
parents: 41951
diff changeset
11 import _multibytecodec
31386
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
12
39030
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
13 ALL_CJKENCODINGS = [
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
14 # _codecs_cn
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
15 'gb2312', 'gbk', 'gb18030', 'hz',
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
16 # _codecs_hk
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
17 'big5hkscs',
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
18 # _codecs_jp
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
19 'cp932', 'shift_jis', 'euc_jp', 'euc_jisx0213', 'shift_jisx0213',
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
20 'euc_jis_2004', 'shift_jis_2004',
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
21 # _codecs_kr
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
22 'cp949', 'euc_kr', 'johab',
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
23 # _codecs_tw
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
24 'big5', 'cp950',
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
25 # _codecs_iso2022
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
26 'iso2022_jp', 'iso2022_jp_1', 'iso2022_jp_2', 'iso2022_jp_2004',
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
27 'iso2022_jp_3', 'iso2022_jp_ext', 'iso2022_kr',
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
28 ]
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
29
37218
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
30 class Test_MultibyteCodec(unittest.TestCase):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
31
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
32 def test_nullcoding(self):
39030
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
33 for enc in ALL_CJKENCODINGS:
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
34 self.assertEqual(''.decode(enc), u'')
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
35 self.assertEqual(unicode('', enc), u'')
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
36 self.assertEqual(u''.encode(enc), '')
37218
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
37
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
38 def test_str_decode(self):
39030
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
39 for enc in ALL_CJKENCODINGS:
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
40 self.assertEqual('abcd'.encode(enc), 'abcd')
37218
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
41
37223
c0818f21f65a Allow long objects as a position value of error callbacks returned.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 37218
diff changeset
42 def test_errorcallback_longindex(self):
c0818f21f65a Allow long objects as a position value of error callbacks returned.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 37218
diff changeset
43 dec = codecs.getdecoder('euc-kr')
c0818f21f65a Allow long objects as a position value of error callbacks returned.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 37218
diff changeset
44 myreplace = lambda exc: (u'', sys.maxint+1)
c0818f21f65a Allow long objects as a position value of error callbacks returned.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 37218
diff changeset
45 codecs.register_error('test.cjktest', myreplace)
c0818f21f65a Allow long objects as a position value of error callbacks returned.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 37218
diff changeset
46 self.assertRaises(IndexError, dec,
c0818f21f65a Allow long objects as a position value of error callbacks returned.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 37218
diff changeset
47 'apple\x92ham\x93spam', 'test.cjktest')
37218
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
48
39029
f93dbfcc663b Add a testcase for r47086 which fixed a bug in codec_getstreamcodec().
Hye-Shik Chang <hyeshik@gmail.com>
parents: 37925
diff changeset
49 def test_codingspec(self):
f93dbfcc663b Add a testcase for r47086 which fixed a bug in codec_getstreamcodec().
Hye-Shik Chang <hyeshik@gmail.com>
parents: 37925
diff changeset
50 try:
39030
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
51 for enc in ALL_CJKENCODINGS:
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
52 print >> open(TESTFN, 'w'), '# coding:', enc
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
53 exec open(TESTFN)
39029
f93dbfcc663b Add a testcase for r47086 which fixed a bug in codec_getstreamcodec().
Hye-Shik Chang <hyeshik@gmail.com>
parents: 37925
diff changeset
54 finally:
f93dbfcc663b Add a testcase for r47086 which fixed a bug in codec_getstreamcodec().
Hye-Shik Chang <hyeshik@gmail.com>
parents: 37925
diff changeset
55 os.unlink(TESTFN)
f93dbfcc663b Add a testcase for r47086 which fixed a bug in codec_getstreamcodec().
Hye-Shik Chang <hyeshik@gmail.com>
parents: 37925
diff changeset
56
47989
efe706aab32b #3305: self->stream can be NULL.
Georg Brandl <georg@python.org>
parents: 41951
diff changeset
57 def test_init_segfault(self):
efe706aab32b #3305: self->stream can be NULL.
Georg Brandl <georg@python.org>
parents: 41951
diff changeset
58 # bug #3305: this used to segfault
efe706aab32b #3305: self->stream can be NULL.
Georg Brandl <georg@python.org>
parents: 41951
diff changeset
59 self.assertRaises(AttributeError,
efe706aab32b #3305: self->stream can be NULL.
Georg Brandl <georg@python.org>
parents: 41951
diff changeset
60 _multibytecodec.MultibyteStreamReader, None)
efe706aab32b #3305: self->stream can be NULL.
Georg Brandl <georg@python.org>
parents: 41951
diff changeset
61 self.assertRaises(AttributeError,
efe706aab32b #3305: self->stream can be NULL.
Georg Brandl <georg@python.org>
parents: 41951
diff changeset
62 _multibytecodec.MultibyteStreamWriter, None)
efe706aab32b #3305: self->stream can be NULL.
Georg Brandl <georg@python.org>
parents: 41951
diff changeset
63
efe706aab32b #3305: self->stream can be NULL.
Georg Brandl <georg@python.org>
parents: 41951
diff changeset
64
37218
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
65 class Test_IncrementalEncoder(unittest.TestCase):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
66
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
67 def test_stateless(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
68 # cp949 encoder isn't stateful at all.
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
69 encoder = codecs.getincrementalencoder('cp949')()
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
70 self.assertEqual(encoder.encode(u'\ud30c\uc774\uc36c \ub9c8\uc744'),
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
71 '\xc6\xc4\xc0\xcc\xbd\xe3 \xb8\xb6\xc0\xbb')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
72 self.assertEqual(encoder.reset(), None)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
73 self.assertEqual(encoder.encode(u'\u2606\u223c\u2606', True),
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
74 '\xa1\xd9\xa1\xad\xa1\xd9')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
75 self.assertEqual(encoder.reset(), None)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
76 self.assertEqual(encoder.encode(u'', True), '')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
77 self.assertEqual(encoder.encode(u'', False), '')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
78 self.assertEqual(encoder.reset(), None)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
79
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
80 def test_stateful(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
81 # jisx0213 encoder is stateful for a few codepoints. eg)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
82 # U+00E6 => A9DC
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
83 # U+00E6 U+0300 => ABC4
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
84 # U+0300 => ABDC
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
85
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
86 encoder = codecs.getincrementalencoder('jisx0213')()
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
87 self.assertEqual(encoder.encode(u'\u00e6\u0300'), '\xab\xc4')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
88 self.assertEqual(encoder.encode(u'\u00e6'), '')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
89 self.assertEqual(encoder.encode(u'\u0300'), '\xab\xc4')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
90 self.assertEqual(encoder.encode(u'\u00e6', True), '\xa9\xdc')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
91
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
92 self.assertEqual(encoder.reset(), None)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
93 self.assertEqual(encoder.encode(u'\u0300'), '\xab\xdc')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
94
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
95 self.assertEqual(encoder.encode(u'\u00e6'), '')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
96 self.assertEqual(encoder.encode('', True), '\xa9\xdc')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
97 self.assertEqual(encoder.encode('', True), '')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
98
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
99 def test_stateful_keep_buffer(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
100 encoder = codecs.getincrementalencoder('jisx0213')()
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
101 self.assertEqual(encoder.encode(u'\u00e6'), '')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
102 self.assertRaises(UnicodeEncodeError, encoder.encode, u'\u0123')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
103 self.assertEqual(encoder.encode(u'\u0300\u00e6'), '\xab\xc4')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
104 self.assertRaises(UnicodeEncodeError, encoder.encode, u'\u0123')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
105 self.assertEqual(encoder.reset(), None)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
106 self.assertEqual(encoder.encode(u'\u0300'), '\xab\xdc')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
107 self.assertEqual(encoder.encode(u'\u00e6'), '')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
108 self.assertRaises(UnicodeEncodeError, encoder.encode, u'\u0123')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
109 self.assertEqual(encoder.encode(u'', True), '\xa9\xdc')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
110
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
111
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
112 class Test_IncrementalDecoder(unittest.TestCase):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
113
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
114 def test_dbcs(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
115 # cp949 decoder is simple with only 1 or 2 bytes sequences.
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
116 decoder = codecs.getincrementaldecoder('cp949')()
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
117 self.assertEqual(decoder.decode('\xc6\xc4\xc0\xcc\xbd'),
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
118 u'\ud30c\uc774')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
119 self.assertEqual(decoder.decode('\xe3 \xb8\xb6\xc0\xbb'),
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
120 u'\uc36c \ub9c8\uc744')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
121 self.assertEqual(decoder.decode(''), u'')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
122
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
123 def test_dbcs_keep_buffer(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
124 decoder = codecs.getincrementaldecoder('cp949')()
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
125 self.assertEqual(decoder.decode('\xc6\xc4\xc0'), u'\ud30c')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
126 self.assertRaises(UnicodeDecodeError, decoder.decode, '', True)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
127 self.assertEqual(decoder.decode('\xcc'), u'\uc774')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
128
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
129 self.assertEqual(decoder.decode('\xc6\xc4\xc0'), u'\ud30c')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
130 self.assertRaises(UnicodeDecodeError, decoder.decode, '\xcc\xbd', True)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
131 self.assertEqual(decoder.decode('\xcc'), u'\uc774')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
132
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
133 def test_iso2022(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
134 decoder = codecs.getincrementaldecoder('iso2022-jp')()
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
135 ESC = '\x1b'
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
136 self.assertEqual(decoder.decode(ESC + '('), u'')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
137 self.assertEqual(decoder.decode('B', True), u'')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
138 self.assertEqual(decoder.decode(ESC + '$'), u'')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
139 self.assertEqual(decoder.decode('B@$'), u'\u4e16')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
140 self.assertEqual(decoder.decode('@$@'), u'\u4e16')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
141 self.assertEqual(decoder.decode('$', True), u'\u4e16')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
142 self.assertEqual(decoder.reset(), None)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
143 self.assertEqual(decoder.decode('@$'), u'@$')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
144 self.assertEqual(decoder.decode(ESC + '$'), u'')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
145 self.assertRaises(UnicodeDecodeError, decoder.decode, '', True)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
146 self.assertEqual(decoder.decode('B@$'), u'\u4e16')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
147
41947
b6c1a1a15a6c Bug #1728403: Fix a bug that CJKCodecs StreamReader hangs when it
Hye-Shik Chang <hyeshik@gmail.com>
parents: 41534
diff changeset
148 class Test_StreamReader(unittest.TestCase):
b6c1a1a15a6c Bug #1728403: Fix a bug that CJKCodecs StreamReader hangs when it
Hye-Shik Chang <hyeshik@gmail.com>
parents: 41534
diff changeset
149 def test_bug1728403(self):
b6c1a1a15a6c Bug #1728403: Fix a bug that CJKCodecs StreamReader hangs when it
Hye-Shik Chang <hyeshik@gmail.com>
parents: 41534
diff changeset
150 try:
b6c1a1a15a6c Bug #1728403: Fix a bug that CJKCodecs StreamReader hangs when it
Hye-Shik Chang <hyeshik@gmail.com>
parents: 41534
diff changeset
151 open(TESTFN, 'w').write('\xa1')
b6c1a1a15a6c Bug #1728403: Fix a bug that CJKCodecs StreamReader hangs when it
Hye-Shik Chang <hyeshik@gmail.com>
parents: 41534
diff changeset
152 f = codecs.open(TESTFN, encoding='cp949')
b6c1a1a15a6c Bug #1728403: Fix a bug that CJKCodecs StreamReader hangs when it
Hye-Shik Chang <hyeshik@gmail.com>
parents: 41534
diff changeset
153 self.assertRaises(UnicodeDecodeError, f.read, 2)
b6c1a1a15a6c Bug #1728403: Fix a bug that CJKCodecs StreamReader hangs when it
Hye-Shik Chang <hyeshik@gmail.com>
parents: 41534
diff changeset
154 finally:
41951
cfbcff66e725 Fix for Windows: close a temporary file before trying to delete it.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 41947
diff changeset
155 try: f.close()
cfbcff66e725 Fix for Windows: close a temporary file before trying to delete it.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 41947
diff changeset
156 except: pass
41947
b6c1a1a15a6c Bug #1728403: Fix a bug that CJKCodecs StreamReader hangs when it
Hye-Shik Chang <hyeshik@gmail.com>
parents: 41534
diff changeset
157 os.unlink(TESTFN)
37218
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
158
31386
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
159 class Test_StreamWriter(unittest.TestCase):
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
160 if len(u'\U00012345') == 2: # UCS2
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
161 def test_gb18030(self):
41947
b6c1a1a15a6c Bug #1728403: Fix a bug that CJKCodecs StreamReader hangs when it
Hye-Shik Chang <hyeshik@gmail.com>
parents: 41534
diff changeset
162 s = StringIO.StringIO()
37218
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
163 c = codecs.getwriter('gb18030')(s)
31386
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
164 c.write(u'123')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
165 self.assertEqual(s.getvalue(), '123')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
166 c.write(u'\U00012345')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
167 self.assertEqual(s.getvalue(), '123\x907\x959')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
168 c.write(u'\U00012345'[0])
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
169 self.assertEqual(s.getvalue(), '123\x907\x959')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
170 c.write(u'\U00012345'[1] + u'\U00012345' + u'\uac00\u00ac')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
171 self.assertEqual(s.getvalue(),
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
172 '123\x907\x959\x907\x959\x907\x959\x827\xcf5\x810\x851')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
173 c.write(u'\U00012345'[0])
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
174 self.assertEqual(s.getvalue(),
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
175 '123\x907\x959\x907\x959\x907\x959\x827\xcf5\x810\x851')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
176 self.assertRaises(UnicodeError, c.reset)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
177 self.assertEqual(s.getvalue(),
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
178 '123\x907\x959\x907\x959\x907\x959\x827\xcf5\x810\x851')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
179
37218
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
180 def test_utf_8(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
181 s= StringIO.StringIO()
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
182 c = codecs.getwriter('utf-8')(s)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
183 c.write(u'123')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
184 self.assertEqual(s.getvalue(), '123')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
185 c.write(u'\U00012345')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
186 self.assertEqual(s.getvalue(), '123\xf0\x92\x8d\x85')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
187
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
188 # Python utf-8 codec can't buffer surrogate pairs yet.
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
189 if 0:
31386
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
190 c.write(u'\U00012345'[0])
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
191 self.assertEqual(s.getvalue(), '123\xf0\x92\x8d\x85')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
192 c.write(u'\U00012345'[1] + u'\U00012345' + u'\uac00\u00ac')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
193 self.assertEqual(s.getvalue(),
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
194 '123\xf0\x92\x8d\x85\xf0\x92\x8d\x85\xf0\x92\x8d\x85'
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
195 '\xea\xb0\x80\xc2\xac')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
196 c.write(u'\U00012345'[0])
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
197 self.assertEqual(s.getvalue(),
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
198 '123\xf0\x92\x8d\x85\xf0\x92\x8d\x85\xf0\x92\x8d\x85'
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
199 '\xea\xb0\x80\xc2\xac')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
200 c.reset()
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
201 self.assertEqual(s.getvalue(),
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
202 '123\xf0\x92\x8d\x85\xf0\x92\x8d\x85\xf0\x92\x8d\x85'
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
203 '\xea\xb0\x80\xc2\xac\xed\xa0\x88')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
204 c.write(u'\U00012345'[1])
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
205 self.assertEqual(s.getvalue(),
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
206 '123\xf0\x92\x8d\x85\xf0\x92\x8d\x85\xf0\x92\x8d\x85'
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
207 '\xea\xb0\x80\xc2\xac\xed\xa0\x88\xed\xbd\x85')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
208
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
209 else: # UCS4
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
210 pass
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
211
32727
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31393
diff changeset
212 def test_streamwriter_strwrite(self):
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31393
diff changeset
213 s = StringIO.StringIO()
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31393
diff changeset
214 wr = codecs.getwriter('gb18030')(s)
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31393
diff changeset
215 wr.write('abcd')
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31393
diff changeset
216 self.assertEqual(s.getvalue(), 'abcd')
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31393
diff changeset
217
36960
3ad42f99a6db Bug #1448490: Fix a bug that ISO-2022 codecs could not handle
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32727
diff changeset
218 class Test_ISO2022(unittest.TestCase):
3ad42f99a6db Bug #1448490: Fix a bug that ISO-2022 codecs could not handle
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32727
diff changeset
219 def test_g2(self):
3ad42f99a6db Bug #1448490: Fix a bug that ISO-2022 codecs could not handle
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32727
diff changeset
220 iso2022jp2 = '\x1b(B:hu4:unit\x1b.A\x1bNi de famille'
3ad42f99a6db Bug #1448490: Fix a bug that ISO-2022 codecs could not handle
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32727
diff changeset
221 uni = u':hu4:unit\xe9 de famille'
3ad42f99a6db Bug #1448490: Fix a bug that ISO-2022 codecs could not handle
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32727
diff changeset
222 self.assertEqual(iso2022jp2.decode('iso2022-jp-2'), uni)
3ad42f99a6db Bug #1448490: Fix a bug that ISO-2022 codecs could not handle
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32727
diff changeset
223
39760
15109781f016 Fix a few bugs on cjkcodecs found by Oren Tirosh:
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39030
diff changeset
224 def test_iso2022_jp_g0(self):
15109781f016 Fix a few bugs on cjkcodecs found by Oren Tirosh:
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39030
diff changeset
225 self.failIf('\x0e' in u'\N{SOFT HYPHEN}'.encode('iso-2022-jp-2'))
15109781f016 Fix a few bugs on cjkcodecs found by Oren Tirosh:
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39030
diff changeset
226 for encoding in ('iso-2022-jp-2004', 'iso-2022-jp-3'):
15109781f016 Fix a few bugs on cjkcodecs found by Oren Tirosh:
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39030
diff changeset
227 e = u'\u3406'.encode(encoding)
15109781f016 Fix a few bugs on cjkcodecs found by Oren Tirosh:
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39030
diff changeset
228 self.failIf(filter(lambda x: x >= '\x80', e))
15109781f016 Fix a few bugs on cjkcodecs found by Oren Tirosh:
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39030
diff changeset
229
40045
347794391ce4 Bug #1572832: fix a bug in ISO-2022 codecs which may cause segfault
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39760
diff changeset
230 def test_bug1572832(self):
347794391ce4 Bug #1572832: fix a bug in ISO-2022 codecs which may cause segfault
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39760
diff changeset
231 if sys.maxunicode >= 0x10000:
347794391ce4 Bug #1572832: fix a bug in ISO-2022 codecs which may cause segfault
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39760
diff changeset
232 myunichr = unichr
347794391ce4 Bug #1572832: fix a bug in ISO-2022 codecs which may cause segfault
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39760
diff changeset
233 else:
347794391ce4 Bug #1572832: fix a bug in ISO-2022 codecs which may cause segfault
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39760
diff changeset
234 myunichr = lambda x: unichr(0xD7C0+(x>>10)) + unichr(0xDC00+(x&0x3FF))
347794391ce4 Bug #1572832: fix a bug in ISO-2022 codecs which may cause segfault
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39760
diff changeset
235
347794391ce4 Bug #1572832: fix a bug in ISO-2022 codecs which may cause segfault
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39760
diff changeset
236 for x in xrange(0x10000, 0x110000):
347794391ce4 Bug #1572832: fix a bug in ISO-2022 codecs which may cause segfault
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39760
diff changeset
237 # Any ISO 2022 codec will cause the segfault
347794391ce4 Bug #1572832: fix a bug in ISO-2022 codecs which may cause segfault
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39760
diff changeset
238 myunichr(x).encode('iso_2022_jp', 'ignore')
347794391ce4 Bug #1572832: fix a bug in ISO-2022 codecs which may cause segfault
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39760
diff changeset
239
31386
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
240 def test_main():
41534
6c4d7678af5e Standardize on test.test_support.run_unittest() (as opposed to a mix of run_unittest() and run_suite()). Also, add functionality to run_unittest() that admits usage of unittest.TestLoader.loadTestsFromModule().
Collin Winter <collinw@gmail.com>
parents: 40045
diff changeset
241 test_support.run_unittest(__name__)
31386
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
242
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
243 if __name__ == "__main__":
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
244 test_main()