Mercurial > cpython-lru_cache-weakref

annotate Lib/test/test_multibytecodec.py @ 79675:c0a2294de787 2.7

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Issue #16199: Backport 2to3 typo fix from 3.2.
author Chris Jerdonek <chris.jerdonek@gmail.com>
date Thu, 11 Oct 2012 16:15:46 -0700
parents e789b4cda872
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
31386
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
1 #!/usr/bin/env python
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
2 #
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
3 # test_multibytecodec.py
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
4 # Unit test for multibytecodec itself
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
5 #
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
6
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
7 from test import test_support
39029
f93dbfcc663b Add a testcase for r47086 which fixed a bug in codec_getstreamcodec().
Hye-Shik Chang <hyeshik@gmail.com>
parents: 37925
diff changeset
8 from test.test_support import TESTFN
f93dbfcc663b Add a testcase for r47086 which fixed a bug in codec_getstreamcodec().
Hye-Shik Chang <hyeshik@gmail.com>
parents: 37925
diff changeset
9 import unittest, StringIO, codecs, sys, os
47989
efe706aab32b #3305: self->stream can be NULL.
Georg Brandl <georg@python.org>
parents: 41951
diff changeset
10 import _multibytecodec
31386
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
11
39030
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
12 ALL_CJKENCODINGS = [
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
13 # _codecs_cn
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
14 'gb2312', 'gbk', 'gb18030', 'hz',
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
15 # _codecs_hk
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
16 'big5hkscs',
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
17 # _codecs_jp
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
18 'cp932', 'shift_jis', 'euc_jp', 'euc_jisx0213', 'shift_jisx0213',
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
19 'euc_jis_2004', 'shift_jis_2004',
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
20 # _codecs_kr
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
21 'cp949', 'euc_kr', 'johab',
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
22 # _codecs_tw
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
23 'big5', 'cp950',
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
24 # _codecs_iso2022
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
25 'iso2022_jp', 'iso2022_jp_1', 'iso2022_jp_2', 'iso2022_jp_2004',
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
26 'iso2022_jp_3', 'iso2022_jp_ext', 'iso2022_kr',
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
27 ]
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
28
37218
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
29 class Test_MultibyteCodec(unittest.TestCase):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
30
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
31 def test_nullcoding(self):
39030
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
32 for enc in ALL_CJKENCODINGS:
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
33 self.assertEqual(''.decode(enc), u'')
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
34 self.assertEqual(unicode('', enc), u'')
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
35 self.assertEqual(u''.encode(enc), '')
37218
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
36
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
37 def test_str_decode(self):
39030
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
38 for enc in ALL_CJKENCODINGS:
58c88a7c9326 Test using all CJK encodings for the testcases which don't require
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39029
diff changeset
39 self.assertEqual('abcd'.encode(enc), 'abcd')
37218
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
40
37223
c0818f21f65a Allow long objects as a position value of error callbacks returned.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 37218
diff changeset
41 def test_errorcallback_longindex(self):
c0818f21f65a Allow long objects as a position value of error callbacks returned.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 37218
diff changeset
42 dec = codecs.getdecoder('euc-kr')
c0818f21f65a Allow long objects as a position value of error callbacks returned.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 37218
diff changeset
43 myreplace = lambda exc: (u'', sys.maxint+1)
c0818f21f65a Allow long objects as a position value of error callbacks returned.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 37218
diff changeset
44 codecs.register_error('test.cjktest', myreplace)
c0818f21f65a Allow long objects as a position value of error callbacks returned.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 37218
diff changeset
45 self.assertRaises(IndexError, dec,
c0818f21f65a Allow long objects as a position value of error callbacks returned.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 37218
diff changeset
46 'apple\x92ham\x93spam', 'test.cjktest')
37218
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
47
39029
f93dbfcc663b Add a testcase for r47086 which fixed a bug in codec_getstreamcodec().
Hye-Shik Chang <hyeshik@gmail.com>
parents: 37925
diff changeset
48 def test_codingspec(self):
65339
fea8f21b199d Merged revisions 85503 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 61843
diff changeset
49 for enc in ALL_CJKENCODINGS:
fea8f21b199d Merged revisions 85503 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 61843
diff changeset
50 code = '# coding: {}\n'.format(enc)
fea8f21b199d Merged revisions 85503 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 61843
diff changeset
51 exec code
39029
f93dbfcc663b Add a testcase for r47086 which fixed a bug in codec_getstreamcodec().
Hye-Shik Chang <hyeshik@gmail.com>
parents: 37925
diff changeset
52
47989
efe706aab32b #3305: self->stream can be NULL.
Georg Brandl <georg@python.org>
parents: 41951
diff changeset
53 def test_init_segfault(self):
efe706aab32b #3305: self->stream can be NULL.
Georg Brandl <georg@python.org>
parents: 41951
diff changeset
54 # bug #3305: this used to segfault
efe706aab32b #3305: self->stream can be NULL.
Georg Brandl <georg@python.org>
parents: 41951
diff changeset
55 self.assertRaises(AttributeError,
efe706aab32b #3305: self->stream can be NULL.
Georg Brandl <georg@python.org>
parents: 41951
diff changeset
56 _multibytecodec.MultibyteStreamReader, None)
efe706aab32b #3305: self->stream can be NULL.
Georg Brandl <georg@python.org>
parents: 41951
diff changeset
57 self.assertRaises(AttributeError,
efe706aab32b #3305: self->stream can be NULL.
Georg Brandl <georg@python.org>
parents: 41951
diff changeset
58 _multibytecodec.MultibyteStreamWriter, None)
efe706aab32b #3305: self->stream can be NULL.
Georg Brandl <georg@python.org>
parents: 41951
diff changeset
59
efe706aab32b #3305: self->stream can be NULL.
Georg Brandl <georg@python.org>
parents: 41951
diff changeset
60
37218
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
61 class Test_IncrementalEncoder(unittest.TestCase):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
62
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
63 def test_stateless(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
64 # cp949 encoder isn't stateful at all.
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
65 encoder = codecs.getincrementalencoder('cp949')()
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
66 self.assertEqual(encoder.encode(u'\ud30c\uc774\uc36c \ub9c8\uc744'),
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
67 '\xc6\xc4\xc0\xcc\xbd\xe3 \xb8\xb6\xc0\xbb')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
68 self.assertEqual(encoder.reset(), None)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
69 self.assertEqual(encoder.encode(u'\u2606\u223c\u2606', True),
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
70 '\xa1\xd9\xa1\xad\xa1\xd9')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
71 self.assertEqual(encoder.reset(), None)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
72 self.assertEqual(encoder.encode(u'', True), '')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
73 self.assertEqual(encoder.encode(u'', False), '')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
74 self.assertEqual(encoder.reset(), None)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
75
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
76 def test_stateful(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
77 # jisx0213 encoder is stateful for a few codepoints. eg)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
78 # U+00E6 => A9DC
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
79 # U+00E6 U+0300 => ABC4
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
80 # U+0300 => ABDC
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
81
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
82 encoder = codecs.getincrementalencoder('jisx0213')()
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
83 self.assertEqual(encoder.encode(u'\u00e6\u0300'), '\xab\xc4')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
84 self.assertEqual(encoder.encode(u'\u00e6'), '')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
85 self.assertEqual(encoder.encode(u'\u0300'), '\xab\xc4')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
86 self.assertEqual(encoder.encode(u'\u00e6', True), '\xa9\xdc')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
87
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
88 self.assertEqual(encoder.reset(), None)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
89 self.assertEqual(encoder.encode(u'\u0300'), '\xab\xdc')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
90
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
91 self.assertEqual(encoder.encode(u'\u00e6'), '')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
92 self.assertEqual(encoder.encode('', True), '\xa9\xdc')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
93 self.assertEqual(encoder.encode('', True), '')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
94
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
95 def test_stateful_keep_buffer(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
96 encoder = codecs.getincrementalencoder('jisx0213')()
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
97 self.assertEqual(encoder.encode(u'\u00e6'), '')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
98 self.assertRaises(UnicodeEncodeError, encoder.encode, u'\u0123')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
99 self.assertEqual(encoder.encode(u'\u0300\u00e6'), '\xab\xc4')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
100 self.assertRaises(UnicodeEncodeError, encoder.encode, u'\u0123')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
101 self.assertEqual(encoder.reset(), None)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
102 self.assertEqual(encoder.encode(u'\u0300'), '\xab\xdc')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
103 self.assertEqual(encoder.encode(u'\u00e6'), '')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
104 self.assertRaises(UnicodeEncodeError, encoder.encode, u'\u0123')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
105 self.assertEqual(encoder.encode(u'', True), '\xa9\xdc')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
106
61845
14d23bddf793 Merged revisions 81454 via svnmerge from
Victor Stinner <victor.stinner@haypocalc.com>
parents: 47989
diff changeset
107 def test_issue5640(self):
14d23bddf793 Merged revisions 81454 via svnmerge from
Victor Stinner <victor.stinner@haypocalc.com>
parents: 47989
diff changeset
108 encoder = codecs.getincrementalencoder('shift-jis')('backslashreplace')
14d23bddf793 Merged revisions 81454 via svnmerge from
Victor Stinner <victor.stinner@haypocalc.com>
parents: 47989
diff changeset
109 self.assertEqual(encoder.encode(u'\xff'), b'\\xff')
14d23bddf793 Merged revisions 81454 via svnmerge from
Victor Stinner <victor.stinner@haypocalc.com>
parents: 47989
diff changeset
110 self.assertEqual(encoder.encode(u'\n'), b'\n')
37218
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
111
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
112 class Test_IncrementalDecoder(unittest.TestCase):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
113
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
114 def test_dbcs(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
115 # cp949 decoder is simple with only 1 or 2 bytes sequences.
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
116 decoder = codecs.getincrementaldecoder('cp949')()
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
117 self.assertEqual(decoder.decode('\xc6\xc4\xc0\xcc\xbd'),
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
118 u'\ud30c\uc774')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
119 self.assertEqual(decoder.decode('\xe3 \xb8\xb6\xc0\xbb'),
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
120 u'\uc36c \ub9c8\uc744')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
121 self.assertEqual(decoder.decode(''), u'')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
122
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
123 def test_dbcs_keep_buffer(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
124 decoder = codecs.getincrementaldecoder('cp949')()
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
125 self.assertEqual(decoder.decode('\xc6\xc4\xc0'), u'\ud30c')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
126 self.assertRaises(UnicodeDecodeError, decoder.decode, '', True)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
127 self.assertEqual(decoder.decode('\xcc'), u'\uc774')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
128
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
129 self.assertEqual(decoder.decode('\xc6\xc4\xc0'), u'\ud30c')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
130 self.assertRaises(UnicodeDecodeError, decoder.decode, '\xcc\xbd', True)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
131 self.assertEqual(decoder.decode('\xcc'), u'\uc774')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
132
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
133 def test_iso2022(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
134 decoder = codecs.getincrementaldecoder('iso2022-jp')()
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
135 ESC = '\x1b'
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
136 self.assertEqual(decoder.decode(ESC + '('), u'')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
137 self.assertEqual(decoder.decode('B', True), u'')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
138 self.assertEqual(decoder.decode(ESC + '$'), u'')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
139 self.assertEqual(decoder.decode('B@$'), u'\u4e16')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
140 self.assertEqual(decoder.decode('@$@'), u'\u4e16')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
141 self.assertEqual(decoder.decode('$', True), u'\u4e16')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
142 self.assertEqual(decoder.reset(), None)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
143 self.assertEqual(decoder.decode('@$'), u'@$')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
144 self.assertEqual(decoder.decode(ESC + '$'), u'')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
145 self.assertRaises(UnicodeDecodeError, decoder.decode, '', True)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
146 self.assertEqual(decoder.decode('B@$'), u'\u4e16')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
147
41949
c5cf9a4f4fa9 (Backport from r55770)
Hye-Shik Chang <hyeshik@gmail.com>
parents: 40045
diff changeset
148 class Test_StreamReader(unittest.TestCase):
c5cf9a4f4fa9 (Backport from r55770)
Hye-Shik Chang <hyeshik@gmail.com>
parents: 40045
diff changeset
149 def test_bug1728403(self):
c5cf9a4f4fa9 (Backport from r55770)
Hye-Shik Chang <hyeshik@gmail.com>
parents: 40045
diff changeset
150 try:
c5cf9a4f4fa9 (Backport from r55770)
Hye-Shik Chang <hyeshik@gmail.com>
parents: 40045
diff changeset
151 open(TESTFN, 'w').write('\xa1')
c5cf9a4f4fa9 (Backport from r55770)
Hye-Shik Chang <hyeshik@gmail.com>
parents: 40045
diff changeset
152 f = codecs.open(TESTFN, encoding='cp949')
c5cf9a4f4fa9 (Backport from r55770)
Hye-Shik Chang <hyeshik@gmail.com>
parents: 40045
diff changeset
153 self.assertRaises(UnicodeDecodeError, f.read, 2)
c5cf9a4f4fa9 (Backport from r55770)
Hye-Shik Chang <hyeshik@gmail.com>
parents: 40045
diff changeset
154 finally:
41952
01813841d64b Fix for Windows: close a temporary file before trying to delete it.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 41949
diff changeset
155 try: f.close()
01813841d64b Fix for Windows: close a temporary file before trying to delete it.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 41949
diff changeset
156 except: pass
41949
c5cf9a4f4fa9 (Backport from r55770)
Hye-Shik Chang <hyeshik@gmail.com>
parents: 40045
diff changeset
157 os.unlink(TESTFN)
37218
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
158
31386
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
159 class Test_StreamWriter(unittest.TestCase):
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
160 if len(u'\U00012345') == 2: # UCS2
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
161 def test_gb18030(self):
41947
b6c1a1a15a6c Bug #1728403: Fix a bug that CJKCodecs StreamReader hangs when it
Hye-Shik Chang <hyeshik@gmail.com>
parents: 41534
diff changeset
162 s = StringIO.StringIO()
37218
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
163 c = codecs.getwriter('gb18030')(s)
31386
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
164 c.write(u'123')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
165 self.assertEqual(s.getvalue(), '123')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
166 c.write(u'\U00012345')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
167 self.assertEqual(s.getvalue(), '123\x907\x959')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
168 c.write(u'\U00012345'[0])
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
169 self.assertEqual(s.getvalue(), '123\x907\x959')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
170 c.write(u'\U00012345'[1] + u'\U00012345' + u'\uac00\u00ac')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
171 self.assertEqual(s.getvalue(),
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
172 '123\x907\x959\x907\x959\x907\x959\x827\xcf5\x810\x851')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
173 c.write(u'\U00012345'[0])
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
174 self.assertEqual(s.getvalue(),
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
175 '123\x907\x959\x907\x959\x907\x959\x827\xcf5\x810\x851')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
176 self.assertRaises(UnicodeError, c.reset)
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
177 self.assertEqual(s.getvalue(),
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
178 '123\x907\x959\x907\x959\x907\x959\x827\xcf5\x810\x851')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
179
37218
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
180 def test_utf_8(self):
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
181 s= StringIO.StringIO()
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
182 c = codecs.getwriter('utf-8')(s)
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
183 c.write(u'123')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
184 self.assertEqual(s.getvalue(), '123')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
185 c.write(u'\U00012345')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
186 self.assertEqual(s.getvalue(), '123\xf0\x92\x8d\x85')
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
187
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
188 # Python utf-8 codec can't buffer surrogate pairs yet.
e234ff8beaa2 Patch #1443155: Add the incremental codecs support for CJK codecs.
Hye-Shik Chang <hyeshik@gmail.com>
parents: 36960
diff changeset
189 if 0:
31386
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
190 c.write(u'\U00012345'[0])
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
191 self.assertEqual(s.getvalue(), '123\xf0\x92\x8d\x85')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
192 c.write(u'\U00012345'[1] + u'\U00012345' + u'\uac00\u00ac')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
193 self.assertEqual(s.getvalue(),
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
194 '123\xf0\x92\x8d\x85\xf0\x92\x8d\x85\xf0\x92\x8d\x85'
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
195 '\xea\xb0\x80\xc2\xac')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
196 c.write(u'\U00012345'[0])
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
197 self.assertEqual(s.getvalue(),
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
198 '123\xf0\x92\x8d\x85\xf0\x92\x8d\x85\xf0\x92\x8d\x85'
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
199 '\xea\xb0\x80\xc2\xac')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
200 c.reset()
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
201 self.assertEqual(s.getvalue(),
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
202 '123\xf0\x92\x8d\x85\xf0\x92\x8d\x85\xf0\x92\x8d\x85'
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
203 '\xea\xb0\x80\xc2\xac\xed\xa0\x88')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
204 c.write(u'\U00012345'[1])
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
205 self.assertEqual(s.getvalue(),
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
206 '123\xf0\x92\x8d\x85\xf0\x92\x8d\x85\xf0\x92\x8d\x85'
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
207 '\xea\xb0\x80\xc2\xac\xed\xa0\x88\xed\xbd\x85')
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
208
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
209 else: # UCS4
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
210 pass
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
211
32727
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31393
diff changeset
212 def test_streamwriter_strwrite(self):
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31393
diff changeset
213 s = StringIO.StringIO()
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31393
diff changeset
214 wr = codecs.getwriter('gb18030')(s)
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31393
diff changeset
215 wr.write('abcd')
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31393
diff changeset
216 self.assertEqual(s.getvalue(), 'abcd')
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source
Hye-Shik Chang <hyeshik@gmail.com>
parents: 31393
diff changeset
217
36960
3ad42f99a6db Bug #1448490: Fix a bug that ISO-2022 codecs could not handle
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32727
diff changeset
218 class Test_ISO2022(unittest.TestCase):
3ad42f99a6db Bug #1448490: Fix a bug that ISO-2022 codecs could not handle
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32727
diff changeset
219 def test_g2(self):
3ad42f99a6db Bug #1448490: Fix a bug that ISO-2022 codecs could not handle
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32727
diff changeset
220 iso2022jp2 = '\x1b(B:hu4:unit\x1b.A\x1bNi de famille'
3ad42f99a6db Bug #1448490: Fix a bug that ISO-2022 codecs could not handle
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32727
diff changeset
221 uni = u':hu4:unit\xe9 de famille'
3ad42f99a6db Bug #1448490: Fix a bug that ISO-2022 codecs could not handle
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32727
diff changeset
222 self.assertEqual(iso2022jp2.decode('iso2022-jp-2'), uni)
3ad42f99a6db Bug #1448490: Fix a bug that ISO-2022 codecs could not handle
Hye-Shik Chang <hyeshik@gmail.com>
parents: 32727
diff changeset
223
39760
15109781f016 Fix a few bugs on cjkcodecs found by Oren Tirosh:
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39030
diff changeset
224 def test_iso2022_jp_g0(self):
58460
17891566a478 use assert[Not]In where appropriate
Ezio Melotti <ezio.melotti@gmail.com>
parents: 55002
diff changeset
225 self.assertNotIn('\x0e', u'\N{SOFT HYPHEN}'.encode('iso-2022-jp-2'))
39760
15109781f016 Fix a few bugs on cjkcodecs found by Oren Tirosh:
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39030
diff changeset
226 for encoding in ('iso-2022-jp-2004', 'iso-2022-jp-3'):
15109781f016 Fix a few bugs on cjkcodecs found by Oren Tirosh:
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39030
diff changeset
227 e = u'\u3406'.encode(encoding)
55002
6768b2350c57 convert usage of fail* to assert*
Benjamin Peterson <benjamin@python.org>
parents: 47989
diff changeset
228 self.assertFalse(filter(lambda x: x >= '\x80', e))
39760
15109781f016 Fix a few bugs on cjkcodecs found by Oren Tirosh:
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39030
diff changeset
229
40045
347794391ce4 Bug #1572832: fix a bug in ISO-2022 codecs which may cause segfault
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39760
diff changeset
230 def test_bug1572832(self):
347794391ce4 Bug #1572832: fix a bug in ISO-2022 codecs which may cause segfault
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39760
diff changeset
231 if sys.maxunicode >= 0x10000:
347794391ce4 Bug #1572832: fix a bug in ISO-2022 codecs which may cause segfault
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39760
diff changeset
232 myunichr = unichr
347794391ce4 Bug #1572832: fix a bug in ISO-2022 codecs which may cause segfault
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39760
diff changeset
233 else:
347794391ce4 Bug #1572832: fix a bug in ISO-2022 codecs which may cause segfault
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39760
diff changeset
234 myunichr = lambda x: unichr(0xD7C0+(x>>10)) + unichr(0xDC00+(x&0x3FF))
347794391ce4 Bug #1572832: fix a bug in ISO-2022 codecs which may cause segfault
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39760
diff changeset
235
347794391ce4 Bug #1572832: fix a bug in ISO-2022 codecs which may cause segfault
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39760
diff changeset
236 for x in xrange(0x10000, 0x110000):
347794391ce4 Bug #1572832: fix a bug in ISO-2022 codecs which may cause segfault
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39760
diff changeset
237 # Any ISO 2022 codec will cause the segfault
347794391ce4 Bug #1572832: fix a bug in ISO-2022 codecs which may cause segfault
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39760
diff changeset
238 myunichr(x).encode('iso_2022_jp', 'ignore')
347794391ce4 Bug #1572832: fix a bug in ISO-2022 codecs which may cause segfault
Hye-Shik Chang <hyeshik@gmail.com>
parents: 39760
diff changeset
239
70350
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
240 class TestStateful(unittest.TestCase):
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
241 text = u'\u4E16\u4E16'
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
242 encoding = 'iso-2022-jp'
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
243 expected = b'\x1b$B@$@$'
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
244 expected_reset = b'\x1b$B@$@$\x1b(B'
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
245
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
246 def test_encode(self):
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
247 self.assertEqual(self.text.encode(self.encoding), self.expected_reset)
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
248
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
249 def test_incrementalencoder(self):
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
250 encoder = codecs.getincrementalencoder(self.encoding)()
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
251 output = b''.join(
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
252 encoder.encode(char)
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
253 for char in self.text)
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
254 self.assertEqual(output, self.expected)
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
255
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
256 def test_incrementalencoder_final(self):
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
257 encoder = codecs.getincrementalencoder(self.encoding)()
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
258 last_index = len(self.text) - 1
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
259 output = b''.join(
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
260 encoder.encode(char, index == last_index)
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
261 for index, char in enumerate(self.text))
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
262 self.assertEqual(output, self.expected_reset)
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
263
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
264 class TestHZStateful(TestStateful):
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
265 text = u'\u804a\u804a'
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
266 encoding = 'hz'
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
267 expected = b'~{ADAD'
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
268 expected_reset = b'~{ADAD~}'
e789b4cda872 Issue #12100: Don't reset incremental encoders of CJK codecs at each call to
Victor Stinner <victor.stinner@haypocalc.com>
parents: 68256
diff changeset
269
31386
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
270 def test_main():
41534
6c4d7678af5e Standardize on test.test_support.run_unittest() (as opposed to a mix of run_unittest() and run_suite()). Also, add functionality to run_unittest() that admits usage of unittest.TestLoader.loadTestsFromModule().
Collin Winter <collinw@gmail.com>
parents: 40045
diff changeset
271 test_support.run_unittest(__name__)
31386
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
272
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
273 if __name__ == "__main__":
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597)
Hye-Shik Chang <hyeshik@gmail.com>
parents:
diff changeset
274 test_main()