cpython-lru_cache-weakref: Lib/test/test

annotate Lib/test/test_codecs.py @ 87690:66c1c9f32567 default tip

Add use_weakref option to functools.lru_cache The use_weakref option makes lru_cache create weak references to the function arguments when creating the cache key, which prevents them from staying in the memory forever even after all other references to them are gone. This is a proof of concept for http://bugs.python.org/issue19859

author	Radomir Dopieralski <devel@sheep.art.pl>
date	Tue, 03 Dec 2013 16:02:48 +0100
parents	694e2708b4a8
children

rev	line source
73579 ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	1 import _testcapi
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	2 import codecs
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	3 import contextlib
73579 ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	4 import io
71489 92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70293 diff changeset	5 import locale
73579 ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	6 import sys
ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	7 import unittest
ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	8 import warnings
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	9 import encodings
73579 ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	10
ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	11 from test import support
72534 85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	12
73138 2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	13 if sys.platform == 'win32':
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	14 VISTA_OR_LATER = (sys.getwindowsversion().major >= 6)
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	15 else:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	16 VISTA_OR_LATER = False
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	17
72683 da3bfa2e9daa Fix text failures when ctypes is not available Antoine Pitrou <solipsis@pitrou.net> parents: 72534 diff changeset	18 try:
da3bfa2e9daa Fix text failures when ctypes is not available Antoine Pitrou <solipsis@pitrou.net> parents: 72534 diff changeset	19 import ctypes
da3bfa2e9daa Fix text failures when ctypes is not available Antoine Pitrou <solipsis@pitrou.net> parents: 72534 diff changeset	20 except ImportError:
da3bfa2e9daa Fix text failures when ctypes is not available Antoine Pitrou <solipsis@pitrou.net> parents: 72534 diff changeset	21 ctypes = None
da3bfa2e9daa Fix text failures when ctypes is not available Antoine Pitrou <solipsis@pitrou.net> parents: 72534 diff changeset	22 SIZEOF_WCHAR_T = -1
da3bfa2e9daa Fix text failures when ctypes is not available Antoine Pitrou <solipsis@pitrou.net> parents: 72534 diff changeset	23 else:
da3bfa2e9daa Fix text failures when ctypes is not available Antoine Pitrou <solipsis@pitrou.net> parents: 72534 diff changeset	24 SIZEOF_WCHAR_T = ctypes.sizeof(ctypes.c_wchar)
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	25
81804 084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	26 def coding_checker(self, coder):
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	27 def check(input, expect):
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	28 self.assertEqual(coder(input), (expect, len(input)))
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	29 return check
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	30
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	31 class Queue(object):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	32 """
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	33 queue: write bytes at one end, read bytes from the other end
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	34 """
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	35 def __init__(self, buffer):
a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	36 self._buffer = buffer
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	37
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	38 def write(self, chars):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	39 self._buffer += chars
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	40
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	41 def read(self, size=-1):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	42 if size<0:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	43 s = self._buffer
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	44 self._buffer = self._buffer[:0] # make empty
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	45 return s
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	46 else:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	47 s = self._buffer[:size]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	48 self._buffer = self._buffer[size:]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	49 return s
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	50
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	51 class MixInCheckStateHandling:
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	52 def check_state_handling_decode(self, encoding, u, s):
41642 a6eb6acfe04a Merged revisions 55007-55179 via svnmerge from Guido van Rossum <guido@python.org> parents: 41606 diff changeset	53 for i in range(len(s)+1):
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	54 d = codecs.getincrementaldecoder(encoding)()
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	55 part1 = d.decode(s[:i])
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	56 state = d.getstate()
58473 71acd4482e65 Merged revisions 77727 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 56036 diff changeset	57 self.assertIsInstance(state[1], int)
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	58 # Check that the condition stated in the documentation for
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	59 # IncrementalDecoder.getstate() holds
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	60 if not state[1]:
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	61 # reset decoder to the default state without anything buffered
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	62 d.setstate((state[0][:0], 0))
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	63 # Feeding the previous input may not produce any output
55593 0bb232078b8c Merged revisions 73715 via svnmerge from Georg Brandl <georg@python.org> parents: 54988 diff changeset	64 self.assertTrue(not d.decode(state[0]))
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	65 # The decoder must return to the same state
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	66 self.assertEqual(state, d.getstate())
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	67 # Create a new decoder and set it to the state
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	68 # we extracted from the old one
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	69 d = codecs.getincrementaldecoder(encoding)()
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	70 d.setstate(state)
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	71 part2 = d.decode(s[i:], True)
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	72 self.assertEqual(u, part1+part2)
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	73
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	74 def check_state_handling_encode(self, encoding, u, s):
41642 a6eb6acfe04a Merged revisions 55007-55179 via svnmerge from Guido van Rossum <guido@python.org> parents: 41606 diff changeset	75 for i in range(len(u)+1):
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	76 d = codecs.getincrementalencoder(encoding)()
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	77 part1 = d.encode(u[:i])
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	78 state = d.getstate()
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	79 d = codecs.getincrementalencoder(encoding)()
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	80 d.setstate(state)
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	81 part2 = d.encode(u[i:], True)
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	82 self.assertEqual(s, part1+part2)
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	83
81383 11e7d0936fa2 #16918: test_codecs now works with unittest test discovery. Patch by Zachary Ware. Ezio Melotti <ezio.melotti@gmail.com> parents: 81382 diff changeset	84 class ReadTest(MixInCheckStateHandling):
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	85 def check_partial(self, input, partialresults):
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	86 # get a StreamReader for the encoding and feed the bytestring version
43912 4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	87 # of input to the reader byte by byte. Read everything available from
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	88 # the StreamReader and check that the results equal the appropriate
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	89 # entries from partialresults.
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	90 q = Queue(b"")
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	91 r = codecs.getreader(self.encoding)(q)
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	92 result = ""
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	93 for (c, partialresult) in zip(input.encode(self.encoding), partialresults):
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	94 q.write(bytes([c]))
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	95 result += r.read()
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	96 self.assertEqual(result, partialresult)
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	97 # check that there's nothing left in the buffers
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	98 self.assertEqual(r.read(), "")
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	99 self.assertEqual(r.bytebuffer, b"")
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	100
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	101 # do the check again, this time using a incremental decoder
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	102 d = codecs.getincrementaldecoder(self.encoding)()
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	103 result = ""
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	104 for (c, partialresult) in zip(input.encode(self.encoding), partialresults):
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	105 result += d.decode(bytes([c]))
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	106 self.assertEqual(result, partialresult)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	107 # check that there's nothing left in the buffers
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	108 self.assertEqual(d.decode(b"", True), "")
a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	109 self.assertEqual(d.buffer, b"")
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	110
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	111 # Check whether the reset method works properly
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	112 d.reset()
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	113 result = ""
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	114 for (c, partialresult) in zip(input.encode(self.encoding), partialresults):
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	115 result += d.decode(bytes([c]))
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	116 self.assertEqual(result, partialresult)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	117 # check that there's nothing left in the buffers
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	118 self.assertEqual(d.decode(b"", True), "")
a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	119 self.assertEqual(d.buffer, b"")
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	120
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	121 # check iterdecode()
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	122 encoded = input.encode(self.encoding)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	123 self.assertEqual(
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	124 input,
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	125 "".join(codecs.iterdecode([bytes([c]) for c in encoded], self.encoding))
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	126 )
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	127
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	128 def test_readline(self):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	129 def getreader(input):
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	130 stream = io.BytesIO(input.encode(self.encoding))
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	131 return codecs.getreader(self.encoding)(stream)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	132
36855 204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	133 def readalllines(input, keepends=True, size=None):
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	134 reader = getreader(input)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	135 lines = []
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	136 while True:
36855 204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	137 line = reader.readline(size=size, keepends=keepends)
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	138 if not line:
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	139 break
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	140 lines.append(line)
36855 204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	141 return "\|".join(lines)
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	142
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	143 s = "foo\nbar\r\nbaz\rspam\u2028eggs"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	144 sexpected = "foo\n\|bar\r\n\|baz\r\|spam\u2028\|eggs"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	145 sexpectednoends = "foo\|bar\|baz\|spam\|eggs"
36855 204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	146 self.assertEqual(readalllines(s, True), sexpected)
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	147 self.assertEqual(readalllines(s, False), sexpectednoends)
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	148 self.assertEqual(readalllines(s, True, 10), sexpected)
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	149 self.assertEqual(readalllines(s, False, 10), sexpectednoends)
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	150
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	151 # Test long lines (multiple calls to read() in readline())
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	152 vw = []
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	153 vwo = []
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	154 for (i, lineend) in enumerate("\n \r\n \r \u2028".split()):
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	155 vw.append((i200)"\3042" + lineend)
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	156 vwo.append((i200)"\3042")
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	157 self.assertEqual(readalllines("".join(vw), True), "".join(vw))
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	158 self.assertEqual(readalllines("".join(vw), False),"".join(vwo))
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	159
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	160 # Test lines where the first read might end with \r, so the
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	161 # reader has to look ahead whether this is a lone \r or a \r\n
41642 a6eb6acfe04a Merged revisions 55007-55179 via svnmerge from Guido van Rossum <guido@python.org> parents: 41606 diff changeset	162 for size in range(80):
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	163 for lineend in "\n \r\n \r \u2028".split():
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	164 s = 10(size"a" + lineend + "xxx\n")
34865 b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	165 reader = getreader(s)
41642 a6eb6acfe04a Merged revisions 55007-55179 via svnmerge from Guido van Rossum <guido@python.org> parents: 41606 diff changeset	166 for i in range(10):
34865 b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	167 self.assertEqual(
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	168 reader.readline(keepends=True),
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	169 size*"a" + lineend,
34865 b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	170 )
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	171 reader = getreader(s)
41642 a6eb6acfe04a Merged revisions 55007-55179 via svnmerge from Guido van Rossum <guido@python.org> parents: 41606 diff changeset	172 for i in range(10):
34865 b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	173 self.assertEqual(
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	174 reader.readline(keepends=False),
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	175 size*"a",
34865 b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	176 )
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	177
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	178 def test_bug1175396(self):
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	179 s = [
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	180 '<%!--===================================================\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	181 ' BLOG index page: show recent articles,\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	182 ' today\'s articles, or articles of a specific date.\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	183 '========================================================--%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	184 '<%@inputencoding="ISO-8859-1"%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	185 '<%@pagetemplate=TEMPLATE.y%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	186 '<%@import=import frog.util, frog%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	187 '<%@import=import frog.objects%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	188 '<%@import=from frog.storageerrors import StorageError%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	189 '<%\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	190 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	191 'import logging\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	192 'log=logging.getLogger("Snakelets.logger")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	193 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	194 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	195 'user=self.SessionCtx.user\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	196 'storageEngine=self.SessionCtx.storageEngine\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	197 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	198 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	199 'def readArticlesFromDate(date, count=None):\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	200 ' entryids=storageEngine.listBlogEntries(date)\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	201 ' entryids.reverse() # descending\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	202 ' if count:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	203 ' entryids=entryids[:count]\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	204 ' try:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	205 ' return [ frog.objects.BlogEntry.load(storageEngine, date, Id) for Id in entryids ]\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	206 ' except StorageError,x:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	207 ' log.error("Error loading articles: "+str(x))\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	208 ' self.abort("cannot load articles")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	209 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	210 'showdate=None\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	211 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	212 'arg=self.Request.getArg()\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	213 'if arg=="today":\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	214 ' #-------------------- TODAY\'S ARTICLES\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	215 ' self.write("<h2>Today\'s articles</h2>")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	216 ' showdate = frog.util.isodatestr() \r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	217 ' entries = readArticlesFromDate(showdate)\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	218 'elif arg=="active":\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	219 ' #-------------------- ACTIVE ARTICLES redirect\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	220 ' self.Yredirect("active.y")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	221 'elif arg=="login":\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	222 ' #-------------------- LOGIN PAGE redirect\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	223 ' self.Yredirect("login.y")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	224 'elif arg=="date":\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	225 ' #-------------------- ARTICLES OF A SPECIFIC DATE\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	226 ' showdate = self.Request.getParameter("date")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	227 ' self.write("<h2>Articles written on %s</h2>"% frog.util.mediumdatestr(showdate))\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	228 ' entries = readArticlesFromDate(showdate)\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	229 'else:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	230 ' #-------------------- RECENT ARTICLES\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	231 ' self.write("<h2>Recent articles</h2>")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	232 ' dates=storageEngine.listBlogEntryDates()\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	233 ' if dates:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	234 ' entries=[]\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	235 ' SHOWAMOUNT=10\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	236 ' for showdate in dates:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	237 ' entries.extend( readArticlesFromDate(showdate, SHOWAMOUNT-len(entries)) )\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	238 ' if len(entries)>=SHOWAMOUNT:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	239 ' break\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	240 ' \r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	241 ]
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	242 stream = io.BytesIO("".join(s).encode(self.encoding))
34865 b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	243 reader = codecs.getreader(self.encoding)(stream)
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	244 for (i, line) in enumerate(reader):
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	245 self.assertEqual(line, s[i])
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	246
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	247 def test_readlinequeue(self):
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	248 q = Queue(b"")
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	249 writer = codecs.getwriter(self.encoding)(q)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	250 reader = codecs.getreader(self.encoding)(q)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	251
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	252 # No lineends
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	253 writer.write("foo\r")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	254 self.assertEqual(reader.readline(keepends=False), "foo")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	255 writer.write("\nbar\r")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	256 self.assertEqual(reader.readline(keepends=False), "")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	257 self.assertEqual(reader.readline(keepends=False), "bar")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	258 writer.write("baz")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	259 self.assertEqual(reader.readline(keepends=False), "baz")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	260 self.assertEqual(reader.readline(keepends=False), "")
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	261
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	262 # Lineends
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	263 writer.write("foo\r")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	264 self.assertEqual(reader.readline(keepends=True), "foo\r")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	265 writer.write("\nbar\r")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	266 self.assertEqual(reader.readline(keepends=True), "\n")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	267 self.assertEqual(reader.readline(keepends=True), "bar\r")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	268 writer.write("baz")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	269 self.assertEqual(reader.readline(keepends=True), "baz")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	270 self.assertEqual(reader.readline(keepends=True), "")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	271 writer.write("foo\r\n")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	272 self.assertEqual(reader.readline(keepends=True), "foo\r\n")
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	273
34384 a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	274 def test_bug1098990_a(self):
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	275 s1 = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy\r\n"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	276 s2 = "offending line: ladfj askldfj klasdj fskla dfzaskdj fasklfj laskd fjasklfzzzzaa%whereisthis!!!\r\n"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	277 s3 = "next line.\r\n"
34384 a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	278
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	279 s = (s1+s2+s3).encode(self.encoding)
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	280 stream = io.BytesIO(s)
34384 a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	281 reader = codecs.getreader(self.encoding)(stream)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	282 self.assertEqual(reader.readline(), s1)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	283 self.assertEqual(reader.readline(), s2)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	284 self.assertEqual(reader.readline(), s3)
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	285 self.assertEqual(reader.readline(), "")
34384 a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	286
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	287 def test_bug1098990_b(self):
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	288 s1 = "aaaaaaaaaaaaaaaaaaaaaaaa\r\n"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	289 s2 = "bbbbbbbbbbbbbbbbbbbbbbbb\r\n"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	290 s3 = "stillokay:bbbbxx\r\n"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	291 s4 = "broken!!!!badbad\r\n"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	292 s5 = "againokay.\r\n"
34384 a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	293
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	294 s = (s1+s2+s3+s4+s5).encode(self.encoding)
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	295 stream = io.BytesIO(s)
34384 a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	296 reader = codecs.getreader(self.encoding)(stream)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	297 self.assertEqual(reader.readline(), s1)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	298 self.assertEqual(reader.readline(), s2)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	299 self.assertEqual(reader.readline(), s3)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	300 self.assertEqual(reader.readline(), s4)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	301 self.assertEqual(reader.readline(), s5)
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	302 self.assertEqual(reader.readline(), "")
34384 a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	303
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	304 ill_formed_sequence_replace = "\ufffd"
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	305
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	306 def test_lone_surrogates(self):
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	307 self.assertRaises(UnicodeEncodeError, "\ud800".encode, self.encoding)
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	308 self.assertEqual("[\uDC80]".encode(self.encoding, "backslashreplace"),
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	309 "[\\udc80]".encode(self.encoding))
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	310 self.assertEqual("[\uDC80]".encode(self.encoding, "xmlcharrefreplace"),
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	311 "[&#56448;]".encode(self.encoding))
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	312 self.assertEqual("[\uDC80]".encode(self.encoding, "ignore"),
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	313 "[]".encode(self.encoding))
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	314 self.assertEqual("[\uDC80]".encode(self.encoding, "replace"),
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	315 "[?]".encode(self.encoding))
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	316
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	317 bom = "".encode(self.encoding)
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	318 for before, after in [("\U00010fff", "A"), ("[", "]"),
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	319 ("A", "\U00010fff")]:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	320 before_sequence = before.encode(self.encoding)[len(bom):]
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	321 after_sequence = after.encode(self.encoding)[len(bom):]
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	322 test_string = before + "\uDC80" + after
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	323 test_sequence = (bom + before_sequence +
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	324 self.ill_formed_sequence + after_sequence)
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	325 self.assertRaises(UnicodeDecodeError, test_sequence.decode,
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	326 self.encoding)
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	327 self.assertEqual(test_string.encode(self.encoding,
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	328 "surrogatepass"),
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	329 test_sequence)
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	330 self.assertEqual(test_sequence.decode(self.encoding,
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	331 "surrogatepass"),
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	332 test_string)
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	333 self.assertEqual(test_sequence.decode(self.encoding, "ignore"),
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	334 before + after)
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	335 self.assertEqual(test_sequence.decode(self.encoding, "replace"),
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	336 before + self.ill_formed_sequence_replace + after)
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	337
81383 11e7d0936fa2 #16918: test_codecs now works with unittest test discovery. Patch by Zachary Ware. Ezio Melotti <ezio.melotti@gmail.com> parents: 81382 diff changeset	338 class UTF32Test(ReadTest, unittest.TestCase):
42626 f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	339 encoding = "utf-32"
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	340 if sys.byteorder == 'little':
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	341 ill_formed_sequence = b"\x80\xdc\x00\x00"
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	342 else:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	343 ill_formed_sequence = b"\x00\x00\xdc\x80"
42626 f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	344
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	345 spamle = (b'\xff\xfe\x00\x00'
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	346 b's\x00\x00\x00p\x00\x00\x00a\x00\x00\x00m\x00\x00\x00'
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	347 b's\x00\x00\x00p\x00\x00\x00a\x00\x00\x00m\x00\x00\x00')
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	348 spambe = (b'\x00\x00\xfe\xff'
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	349 b'\x00\x00\x00s\x00\x00\x00p\x00\x00\x00a\x00\x00\x00m'
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	350 b'\x00\x00\x00s\x00\x00\x00p\x00\x00\x00a\x00\x00\x00m')
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	351
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	352 def test_only_one_bom(self):
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	353 _,_,reader,writer = codecs.lookup(self.encoding)
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	354 # encode some stream
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	355 s = io.BytesIO()
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	356 f = writer(s)
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	357 f.write("spam")
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	358 f.write("spam")
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	359 d = s.getvalue()
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	360 # check whether there is exactly one BOM in it
55593 0bb232078b8c Merged revisions 73715 via svnmerge from Georg Brandl <georg@python.org> parents: 54988 diff changeset	361 self.assertTrue(d == self.spamle or d == self.spambe)
42626 f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	362 # try to read it back
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	363 s = io.BytesIO(d)
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	364 f = reader(s)
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	365 self.assertEqual(f.read(), "spamspam")
42626 f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	366
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	367 def test_badbom(self):
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	368 s = io.BytesIO(4*b"\xff")
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	369 f = codecs.getreader(self.encoding)(s)
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	370 self.assertRaises(UnicodeError, f.read)
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	371
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	372 s = io.BytesIO(8*b"\xff")
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	373 f = codecs.getreader(self.encoding)(s)
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	374 self.assertRaises(UnicodeError, f.read)
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	375
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	376 def test_partial(self):
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	377 self.check_partial(
81329 4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	378 "\x00\xff\u0100\uffff\U00010000",
42626 f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	379 [
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	380 "", # first byte of BOM read
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	381 "", # second byte of BOM read
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	382 "", # third byte of BOM read
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	383 "", # fourth byte of BOM read => byteorder known
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	384 "",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	385 "",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	386 "",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	387 "\x00",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	388 "\x00",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	389 "\x00",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	390 "\x00",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	391 "\x00\xff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	392 "\x00\xff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	393 "\x00\xff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	394 "\x00\xff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	395 "\x00\xff\u0100",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	396 "\x00\xff\u0100",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	397 "\x00\xff\u0100",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	398 "\x00\xff\u0100",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	399 "\x00\xff\u0100\uffff",
81329 4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	400 "\x00\xff\u0100\uffff",
4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	401 "\x00\xff\u0100\uffff",
4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	402 "\x00\xff\u0100\uffff",
4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	403 "\x00\xff\u0100\uffff\U00010000",
42626 f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	404 ]
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	405 )
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	406
56036 a9809f3b164f Merged revisions 74871 via svnmerge from Georg Brandl <georg@python.org> parents: 55593 diff changeset	407 def test_handlers(self):
a9809f3b164f Merged revisions 74871 via svnmerge from Georg Brandl <georg@python.org> parents: 55593 diff changeset	408 self.assertEqual(('\ufffd', 1),
a9809f3b164f Merged revisions 74871 via svnmerge from Georg Brandl <georg@python.org> parents: 55593 diff changeset	409 codecs.utf_32_decode(b'\x01', 'replace', True))
a9809f3b164f Merged revisions 74871 via svnmerge from Georg Brandl <georg@python.org> parents: 55593 diff changeset	410 self.assertEqual(('', 1),
a9809f3b164f Merged revisions 74871 via svnmerge from Georg Brandl <georg@python.org> parents: 55593 diff changeset	411 codecs.utf_32_decode(b'\x01', 'ignore', True))
a9809f3b164f Merged revisions 74871 via svnmerge from Georg Brandl <georg@python.org> parents: 55593 diff changeset	412
42626 f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	413 def test_errors(self):
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	414 self.assertRaises(UnicodeDecodeError, codecs.utf_32_decode,
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	415 b"\xff", "strict", True)
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	416
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	417 def test_decoder_state(self):
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	418 self.check_state_handling_decode(self.encoding,
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	419 "spamspam", self.spamle)
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	420 self.check_state_handling_decode(self.encoding,
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	421 "spamspam", self.spambe)
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	422
62258 8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	423 def test_issue8941(self):
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	424 # Issue #8941: insufficient result allocation when decoding into
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	425 # surrogate pairs on UCS-2 builds.
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	426 encoded_le = b'\xff\xfe\x00\x00' + b'\x00\x00\x01\x00' * 1024
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	427 self.assertEqual('\U00010000' * 1024,
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	428 codecs.utf_32_decode(encoded_le)[0])
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	429 encoded_be = b'\x00\x00\xfe\xff' + b'\x00\x01\x00\x00' * 1024
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	430 self.assertEqual('\U00010000' * 1024,
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	431 codecs.utf_32_decode(encoded_be)[0])
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	432
81383 11e7d0936fa2 #16918: test_codecs now works with unittest test discovery. Patch by Zachary Ware. Ezio Melotti <ezio.melotti@gmail.com> parents: 81382 diff changeset	433 class UTF32LETest(ReadTest, unittest.TestCase):
42626 f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	434 encoding = "utf-32-le"
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	435 ill_formed_sequence = b"\x80\xdc\x00\x00"
42626 f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	436
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	437 def test_partial(self):
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	438 self.check_partial(
81329 4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	439 "\x00\xff\u0100\uffff\U00010000",
42626 f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	440 [
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	441 "",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	442 "",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	443 "",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	444 "\x00",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	445 "\x00",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	446 "\x00",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	447 "\x00",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	448 "\x00\xff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	449 "\x00\xff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	450 "\x00\xff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	451 "\x00\xff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	452 "\x00\xff\u0100",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	453 "\x00\xff\u0100",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	454 "\x00\xff\u0100",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	455 "\x00\xff\u0100",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	456 "\x00\xff\u0100\uffff",
81329 4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	457 "\x00\xff\u0100\uffff",
4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	458 "\x00\xff\u0100\uffff",
4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	459 "\x00\xff\u0100\uffff",
4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	460 "\x00\xff\u0100\uffff\U00010000",
42626 f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	461 ]
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	462 )
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	463
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	464 def test_simple(self):
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	465 self.assertEqual("\U00010203".encode(self.encoding), b"\x03\x02\x01\x00")
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	466
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	467 def test_errors(self):
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	468 self.assertRaises(UnicodeDecodeError, codecs.utf_32_le_decode,
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	469 b"\xff", "strict", True)
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	470
62258 8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	471 def test_issue8941(self):
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	472 # Issue #8941: insufficient result allocation when decoding into
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	473 # surrogate pairs on UCS-2 builds.
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	474 encoded = b'\x00\x00\x01\x00' * 1024
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	475 self.assertEqual('\U00010000' * 1024,
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	476 codecs.utf_32_le_decode(encoded)[0])
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	477
81383 11e7d0936fa2 #16918: test_codecs now works with unittest test discovery. Patch by Zachary Ware. Ezio Melotti <ezio.melotti@gmail.com> parents: 81382 diff changeset	478 class UTF32BETest(ReadTest, unittest.TestCase):
42626 f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	479 encoding = "utf-32-be"
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	480 ill_formed_sequence = b"\x00\x00\xdc\x80"
42626 f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	481
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	482 def test_partial(self):
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	483 self.check_partial(
81329 4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	484 "\x00\xff\u0100\uffff\U00010000",
42626 f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	485 [
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	486 "",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	487 "",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	488 "",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	489 "\x00",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	490 "\x00",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	491 "\x00",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	492 "\x00",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	493 "\x00\xff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	494 "\x00\xff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	495 "\x00\xff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	496 "\x00\xff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	497 "\x00\xff\u0100",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	498 "\x00\xff\u0100",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	499 "\x00\xff\u0100",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	500 "\x00\xff\u0100",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	501 "\x00\xff\u0100\uffff",
81329 4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	502 "\x00\xff\u0100\uffff",
4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	503 "\x00\xff\u0100\uffff",
4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	504 "\x00\xff\u0100\uffff",
4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	505 "\x00\xff\u0100\uffff\U00010000",
42626 f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	506 ]
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	507 )
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	508
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	509 def test_simple(self):
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	510 self.assertEqual("\U00010203".encode(self.encoding), b"\x00\x01\x02\x03")
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	511
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	512 def test_errors(self):
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	513 self.assertRaises(UnicodeDecodeError, codecs.utf_32_be_decode,
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	514 b"\xff", "strict", True)
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	515
62258 8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	516 def test_issue8941(self):
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	517 # Issue #8941: insufficient result allocation when decoding into
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	518 # surrogate pairs on UCS-2 builds.
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	519 encoded = b'\x00\x01\x00\x00' * 1024
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	520 self.assertEqual('\U00010000' * 1024,
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	521 codecs.utf_32_be_decode(encoded)[0])
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	522
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	523
81383 11e7d0936fa2 #16918: test_codecs now works with unittest test discovery. Patch by Zachary Ware. Ezio Melotti <ezio.melotti@gmail.com> parents: 81382 diff changeset	524 class UTF16Test(ReadTest, unittest.TestCase):
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	525 encoding = "utf-16"
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	526 if sys.byteorder == 'little':
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	527 ill_formed_sequence = b"\x80\xdc"
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	528 else:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	529 ill_formed_sequence = b"\xdc\x80"
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	530
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	531 spamle = b'\xff\xfes\x00p\x00a\x00m\x00s\x00p\x00a\x00m\x00'
a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	532 spambe = b'\xfe\xff\x00s\x00p\x00a\x00m\x00s\x00p\x00a\x00m'
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	533
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	534 def test_only_one_bom(self):
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	535 _,_,reader,writer = codecs.lookup(self.encoding)
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	536 # encode some stream
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	537 s = io.BytesIO()
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	538 f = writer(s)
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	539 f.write("spam")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	540 f.write("spam")
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	541 d = s.getvalue()
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	542 # check whether there is exactly one BOM in it
55593 0bb232078b8c Merged revisions 73715 via svnmerge from Georg Brandl <georg@python.org> parents: 54988 diff changeset	543 self.assertTrue(d == self.spamle or d == self.spambe)
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	544 # try to read it back
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	545 s = io.BytesIO(d)
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	546 f = reader(s)
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	547 self.assertEqual(f.read(), "spamspam")
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	548
34538 37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	549 def test_badbom(self):
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	550 s = io.BytesIO(b"\xff\xff")
34562 7a1277fad04e Fix stupid typo: Don't read from a writer. Walter Dörwald <walter@livinglogic.de> parents: 34538 diff changeset	551 f = codecs.getreader(self.encoding)(s)
34538 37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	552 self.assertRaises(UnicodeError, f.read)
37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	553
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	554 s = io.BytesIO(b"\xff\xff\xff\xff")
34562 7a1277fad04e Fix stupid typo: Don't read from a writer. Walter Dörwald <walter@livinglogic.de> parents: 34538 diff changeset	555 f = codecs.getreader(self.encoding)(s)
34538 37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	556 self.assertRaises(UnicodeError, f.read)
37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	557
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	558 def test_partial(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	559 self.check_partial(
81329 4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	560 "\x00\xff\u0100\uffff\U00010000",
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	561 [
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	562 "", # first byte of BOM read
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	563 "", # second byte of BOM read => byteorder known
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	564 "",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	565 "\x00",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	566 "\x00",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	567 "\x00\xff",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	568 "\x00\xff",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	569 "\x00\xff\u0100",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	570 "\x00\xff\u0100",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	571 "\x00\xff\u0100\uffff",
81329 4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	572 "\x00\xff\u0100\uffff",
4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	573 "\x00\xff\u0100\uffff",
4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	574 "\x00\xff\u0100\uffff",
4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	575 "\x00\xff\u0100\uffff\U00010000",
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	576 ]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	577 )
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	578
56036 a9809f3b164f Merged revisions 74871 via svnmerge from Georg Brandl <georg@python.org> parents: 55593 diff changeset	579 def test_handlers(self):
a9809f3b164f Merged revisions 74871 via svnmerge from Georg Brandl <georg@python.org> parents: 55593 diff changeset	580 self.assertEqual(('\ufffd', 1),
a9809f3b164f Merged revisions 74871 via svnmerge from Georg Brandl <georg@python.org> parents: 55593 diff changeset	581 codecs.utf_16_decode(b'\x01', 'replace', True))
a9809f3b164f Merged revisions 74871 via svnmerge from Georg Brandl <georg@python.org> parents: 55593 diff changeset	582 self.assertEqual(('', 1),
a9809f3b164f Merged revisions 74871 via svnmerge from Georg Brandl <georg@python.org> parents: 55593 diff changeset	583 codecs.utf_16_decode(b'\x01', 'ignore', True))
a9809f3b164f Merged revisions 74871 via svnmerge from Georg Brandl <georg@python.org> parents: 55593 diff changeset	584
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	585 def test_errors(self):
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	586 self.assertRaises(UnicodeDecodeError, codecs.utf_16_decode,
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	587 b"\xff", "strict", True)
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	588
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	589 def test_decoder_state(self):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	590 self.check_state_handling_decode(self.encoding,
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	591 "spamspam", self.spamle)
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	592 self.check_state_handling_decode(self.encoding,
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	593 "spamspam", self.spambe)
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	594
59171 0d590e5e840c Recorded merge of revisions 78462,78484 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 56036 diff changeset	595 def test_bug691291(self):
0d590e5e840c Recorded merge of revisions 78462,78484 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 56036 diff changeset	596 # Files are always opened in binary mode, even if no binary mode was
0d590e5e840c Recorded merge of revisions 78462,78484 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 56036 diff changeset	597 # specified. This means that no automatic conversion of '\n' is done
0d590e5e840c Recorded merge of revisions 78462,78484 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 56036 diff changeset	598 # on reading and writing.
0d590e5e840c Recorded merge of revisions 78462,78484 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 56036 diff changeset	599 s1 = 'Hello\r\nworld\r\n'
0d590e5e840c Recorded merge of revisions 78462,78484 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 56036 diff changeset	600
0d590e5e840c Recorded merge of revisions 78462,78484 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 56036 diff changeset	601 s = s1.encode(self.encoding)
70292 b538445a9df8 test_codecs now removes the temporay file (created by the test) Victor Stinner <victor.stinner@haypocalc.com> parents: 66340 diff changeset	602 self.addCleanup(support.unlink, support.TESTFN)
b538445a9df8 test_codecs now removes the temporay file (created by the test) Victor Stinner <victor.stinner@haypocalc.com> parents: 66340 diff changeset	603 with open(support.TESTFN, 'wb') as fp:
b538445a9df8 test_codecs now removes the temporay file (created by the test) Victor Stinner <victor.stinner@haypocalc.com> parents: 66340 diff changeset	604 fp.write(s)
87520 694e2708b4a8 Issue #15204: Silence and check the 'U' mode deprecation warnings in tests. Serhiy Storchaka <storchaka@gmail.com> parents: 87430 diff changeset	605 with support.check_warnings(('', DeprecationWarning)):
694e2708b4a8 Issue #15204: Silence and check the 'U' mode deprecation warnings in tests. Serhiy Storchaka <storchaka@gmail.com> parents: 87430 diff changeset	606 reader = codecs.open(support.TESTFN, 'U', encoding=self.encoding)
694e2708b4a8 Issue #15204: Silence and check the 'U' mode deprecation warnings in tests. Serhiy Storchaka <storchaka@gmail.com> parents: 87430 diff changeset	607 with reader:
70292 b538445a9df8 test_codecs now removes the temporay file (created by the test) Victor Stinner <victor.stinner@haypocalc.com> parents: 66340 diff changeset	608 self.assertEqual(reader.read(), s1)
59171 0d590e5e840c Recorded merge of revisions 78462,78484 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 56036 diff changeset	609
81383 11e7d0936fa2 #16918: test_codecs now works with unittest test discovery. Patch by Zachary Ware. Ezio Melotti <ezio.melotti@gmail.com> parents: 81382 diff changeset	610 class UTF16LETest(ReadTest, unittest.TestCase):
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	611 encoding = "utf-16-le"
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	612 ill_formed_sequence = b"\x80\xdc"
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	613
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	614 def test_partial(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	615 self.check_partial(
81329 4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	616 "\x00\xff\u0100\uffff\U00010000",
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	617 [
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	618 "",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	619 "\x00",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	620 "\x00",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	621 "\x00\xff",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	622 "\x00\xff",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	623 "\x00\xff\u0100",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	624 "\x00\xff\u0100",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	625 "\x00\xff\u0100\uffff",
81329 4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	626 "\x00\xff\u0100\uffff",
4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	627 "\x00\xff\u0100\uffff",
4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	628 "\x00\xff\u0100\uffff",
4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	629 "\x00\xff\u0100\uffff\U00010000",
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	630 ]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	631 )
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	632
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	633 def test_errors(self):
78206 034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	634 tests = [
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	635 (b'\xff', '\ufffd'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	636 (b'A\x00Z', 'A\ufffd'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	637 (b'A\x00B\x00C\x00D\x00Z', 'ABCD\ufffd'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	638 (b'\x00\xd8', '\ufffd'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	639 (b'\x00\xd8A', '\ufffd'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	640 (b'\x00\xd8A\x00', '\ufffdA'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	641 (b'\x00\xdcA\x00', '\ufffdA'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	642 ]
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	643 for raw, expected in tests:
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	644 self.assertRaises(UnicodeDecodeError, codecs.utf_16_le_decode,
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	645 raw, 'strict', True)
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	646 self.assertEqual(raw.decode('utf-16le', 'replace'), expected)
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	647
66797 113925b79b97 Issue #10546: UTF-16-LE and UTF-16-BE do support non-BMP characters Victor Stinner <victor.stinner@haypocalc.com> parents: 66605 diff changeset	648 def test_nonbmp(self):
113925b79b97 Issue #10546: UTF-16-LE and UTF-16-BE do support non-BMP characters Victor Stinner <victor.stinner@haypocalc.com> parents: 66605 diff changeset	649 self.assertEqual("\U00010203".encode(self.encoding),
113925b79b97 Issue #10546: UTF-16-LE and UTF-16-BE do support non-BMP characters Victor Stinner <victor.stinner@haypocalc.com> parents: 66605 diff changeset	650 b'\x00\xd8\x03\xde')
113925b79b97 Issue #10546: UTF-16-LE and UTF-16-BE do support non-BMP characters Victor Stinner <victor.stinner@haypocalc.com> parents: 66605 diff changeset	651 self.assertEqual(b'\x00\xd8\x03\xde'.decode(self.encoding),
113925b79b97 Issue #10546: UTF-16-LE and UTF-16-BE do support non-BMP characters Victor Stinner <victor.stinner@haypocalc.com> parents: 66605 diff changeset	652 "\U00010203")
113925b79b97 Issue #10546: UTF-16-LE and UTF-16-BE do support non-BMP characters Victor Stinner <victor.stinner@haypocalc.com> parents: 66605 diff changeset	653
81383 11e7d0936fa2 #16918: test_codecs now works with unittest test discovery. Patch by Zachary Ware. Ezio Melotti <ezio.melotti@gmail.com> parents: 81382 diff changeset	654 class UTF16BETest(ReadTest, unittest.TestCase):
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	655 encoding = "utf-16-be"
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	656 ill_formed_sequence = b"\xdc\x80"
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	657
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	658 def test_partial(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	659 self.check_partial(
81329 4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	660 "\x00\xff\u0100\uffff\U00010000",
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	661 [
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	662 "",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	663 "\x00",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	664 "\x00",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	665 "\x00\xff",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	666 "\x00\xff",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	667 "\x00\xff\u0100",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	668 "\x00\xff\u0100",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	669 "\x00\xff\u0100\uffff",
81329 4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	670 "\x00\xff\u0100\uffff",
4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	671 "\x00\xff\u0100\uffff",
4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	672 "\x00\xff\u0100\uffff",
4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	673 "\x00\xff\u0100\uffff\U00010000",
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	674 ]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	675 )
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	676
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	677 def test_errors(self):
78206 034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	678 tests = [
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	679 (b'\xff', '\ufffd'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	680 (b'\x00A\xff', 'A\ufffd'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	681 (b'\x00A\x00B\x00C\x00DZ', 'ABCD\ufffd'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	682 (b'\xd8\x00', '\ufffd'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	683 (b'\xd8\x00\xdc', '\ufffd'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	684 (b'\xd8\x00\x00A', '\ufffdA'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	685 (b'\xdc\x00\x00A', '\ufffdA'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	686 ]
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	687 for raw, expected in tests:
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	688 self.assertRaises(UnicodeDecodeError, codecs.utf_16_be_decode,
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	689 raw, 'strict', True)
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	690 self.assertEqual(raw.decode('utf-16be', 'replace'), expected)
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	691
66797 113925b79b97 Issue #10546: UTF-16-LE and UTF-16-BE do support non-BMP characters Victor Stinner <victor.stinner@haypocalc.com> parents: 66605 diff changeset	692 def test_nonbmp(self):
113925b79b97 Issue #10546: UTF-16-LE and UTF-16-BE do support non-BMP characters Victor Stinner <victor.stinner@haypocalc.com> parents: 66605 diff changeset	693 self.assertEqual("\U00010203".encode(self.encoding),
113925b79b97 Issue #10546: UTF-16-LE and UTF-16-BE do support non-BMP characters Victor Stinner <victor.stinner@haypocalc.com> parents: 66605 diff changeset	694 b'\xd8\x00\xde\x03')
113925b79b97 Issue #10546: UTF-16-LE and UTF-16-BE do support non-BMP characters Victor Stinner <victor.stinner@haypocalc.com> parents: 66605 diff changeset	695 self.assertEqual(b'\xd8\x00\xde\x03'.decode(self.encoding),
113925b79b97 Issue #10546: UTF-16-LE and UTF-16-BE do support non-BMP characters Victor Stinner <victor.stinner@haypocalc.com> parents: 66605 diff changeset	696 "\U00010203")
113925b79b97 Issue #10546: UTF-16-LE and UTF-16-BE do support non-BMP characters Victor Stinner <victor.stinner@haypocalc.com> parents: 66605 diff changeset	697
81383 11e7d0936fa2 #16918: test_codecs now works with unittest test discovery. Patch by Zachary Ware. Ezio Melotti <ezio.melotti@gmail.com> parents: 81382 diff changeset	698 class UTF8Test(ReadTest, unittest.TestCase):
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	699 encoding = "utf-8"
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	700 ill_formed_sequence = b"\xed\xb2\x80"
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	701 ill_formed_sequence_replace = "\ufffd" * 3
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	702
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	703 def test_partial(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	704 self.check_partial(
81329 4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	705 "\x00\xff\u07ff\u0800\uffff\U00010000",
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	706 [
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	707 "\x00",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	708 "\x00",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	709 "\x00\xff",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	710 "\x00\xff",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	711 "\x00\xff\u07ff",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	712 "\x00\xff\u07ff",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	713 "\x00\xff\u07ff",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	714 "\x00\xff\u07ff\u0800",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	715 "\x00\xff\u07ff\u0800",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	716 "\x00\xff\u07ff\u0800",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	717 "\x00\xff\u07ff\u0800\uffff",
81329 4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	718 "\x00\xff\u07ff\u0800\uffff",
4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	719 "\x00\xff\u07ff\u0800\uffff",
4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	720 "\x00\xff\u07ff\u0800\uffff",
4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	721 "\x00\xff\u07ff\u0800\uffff\U00010000",
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	722 ]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	723 )
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	724
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	725 def test_decoder_state(self):
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	726 u = "\x00\x7f\x80\xff\u0100\u07ff\u0800\uffff\U0010ffff"
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	727 self.check_state_handling_decode(self.encoding,
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	728 u, u.encode(self.encoding))
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	729
53681 52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 50975 diff changeset	730 def test_lone_surrogates(self):
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	731 super().test_lone_surrogates()
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	732 # not sure if this is making sense for
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	733 # UTF-16 and UTF-32
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	734 self.assertEqual("[\uDC80]".encode('utf-8', "surrogateescape"),
60875 149badf73bca Merged revisions 80382 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 59171 diff changeset	735 b'[\x80]')
53681 52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 50975 diff changeset	736
53965 fbcdae3e0ca1 Rename the surrogates error handler to surrogatepass. Martin v. Löwis <martin@v.loewis.de> parents: 53859 diff changeset	737 def test_surrogatepass_handler(self):
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	738 self.assertEqual("abc\ud800def".encode("utf-8", "surrogatepass"),
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	739 b"abc\xed\xa0\x80def")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	740 self.assertEqual(b"abc\xed\xa0\x80def".decode("utf-8", "surrogatepass"),
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	741 "abc\ud800def")
72475 8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71512 diff changeset	742 self.assertEqual("\U00010fff\uD800".encode("utf-8", "surrogatepass"),
8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71512 diff changeset	743 b"\xf0\x90\xbf\xbf\xed\xa0\x80")
8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71512 diff changeset	744 self.assertEqual(b"\xf0\x90\xbf\xbf\xed\xa0\x80".decode("utf-8", "surrogatepass"),
8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71512 diff changeset	745 "\U00010fff\uD800")
53965 fbcdae3e0ca1 Rename the surrogates error handler to surrogatepass. Martin v. Löwis <martin@v.loewis.de> parents: 53859 diff changeset	746 self.assertTrue(codecs.lookup_error("surrogatepass"))
79953 74d65c746f63 bounds check for bad data (thanks amaury) Philip Jenvey <pjenvey@underboss.org> parents: 79107 diff changeset	747 with self.assertRaises(UnicodeDecodeError):
74d65c746f63 bounds check for bad data (thanks amaury) Philip Jenvey <pjenvey@underboss.org> parents: 79107 diff changeset	748 b"abc\xed\xa0".decode("utf-8", "surrogatepass")
80216 18ed608ee594 #16336: fix input checking in the surrogatepass error handler. Patch by Serhiy Storchaka. Ezio Melotti <ezio.melotti@gmail.com> parents: 79953 diff changeset	749 with self.assertRaises(UnicodeDecodeError):
18ed608ee594 #16336: fix input checking in the surrogatepass error handler. Patch by Serhiy Storchaka. Ezio Melotti <ezio.melotti@gmail.com> parents: 79953 diff changeset	750 b"abc\xed\xa0z".decode("utf-8", "surrogatepass")
53681 52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 50975 diff changeset	751
73138 2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	752 @unittest.skipUnless(sys.platform == 'win32',
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	753 'cp65001 is a Windows-only codec')
81383 11e7d0936fa2 #16918: test_codecs now works with unittest test discovery. Patch by Zachary Ware. Ezio Melotti <ezio.melotti@gmail.com> parents: 81382 diff changeset	754 class CP65001Test(ReadTest, unittest.TestCase):
73138 2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	755 encoding = "cp65001"
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	756
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	757 def test_encode(self):
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	758 tests = [
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	759 ('abc', 'strict', b'abc'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	760 ('\xe9\u20ac', 'strict', b'\xc3\xa9\xe2\x82\xac'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	761 ('\U0010ffff', 'strict', b'\xf4\x8f\xbf\xbf'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	762 ]
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	763 if VISTA_OR_LATER:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	764 tests.extend((
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	765 ('\udc80', 'strict', None),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	766 ('\udc80', 'ignore', b''),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	767 ('\udc80', 'replace', b'?'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	768 ('\udc80', 'backslashreplace', b'\\udc80'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	769 ('\udc80', 'surrogatepass', b'\xed\xb2\x80'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	770 ))
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	771 else:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	772 tests.append(('\udc80', 'strict', b'\xed\xb2\x80'))
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	773 for text, errors, expected in tests:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	774 if expected is not None:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	775 try:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	776 encoded = text.encode('cp65001', errors)
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	777 except UnicodeEncodeError as err:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	778 self.fail('Unable to encode %a to cp65001 with '
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	779 'errors=%r: %s' % (text, errors, err))
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	780 self.assertEqual(encoded, expected,
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	781 '%a.encode("cp65001", %r)=%a != %a'
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	782 % (text, errors, encoded, expected))
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	783 else:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	784 self.assertRaises(UnicodeEncodeError,
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	785 text.encode, "cp65001", errors)
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	786
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	787 def test_decode(self):
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	788 tests = [
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	789 (b'abc', 'strict', 'abc'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	790 (b'\xc3\xa9\xe2\x82\xac', 'strict', '\xe9\u20ac'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	791 (b'\xf4\x8f\xbf\xbf', 'strict', '\U0010ffff'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	792 (b'\xef\xbf\xbd', 'strict', '\ufffd'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	793 (b'[\xc3\xa9]', 'strict', '[\xe9]'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	794 # invalid bytes
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	795 (b'[\xff]', 'strict', None),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	796 (b'[\xff]', 'ignore', '[]'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	797 (b'[\xff]', 'replace', '[\ufffd]'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	798 (b'[\xff]', 'surrogateescape', '[\udcff]'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	799 ]
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	800 if VISTA_OR_LATER:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	801 tests.extend((
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	802 (b'[\xed\xb2\x80]', 'strict', None),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	803 (b'[\xed\xb2\x80]', 'ignore', '[]'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	804 (b'[\xed\xb2\x80]', 'replace', '[\ufffd\ufffd\ufffd]'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	805 ))
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	806 else:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	807 tests.extend((
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	808 (b'[\xed\xb2\x80]', 'strict', '[\udc80]'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	809 ))
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	810 for raw, errors, expected in tests:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	811 if expected is not None:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	812 try:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	813 decoded = raw.decode('cp65001', errors)
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	814 except UnicodeDecodeError as err:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	815 self.fail('Unable to decode %a from cp65001 with '
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	816 'errors=%r: %s' % (raw, errors, err))
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	817 self.assertEqual(decoded, expected,
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	818 '%a.decode("cp65001", %r)=%a != %a'
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	819 % (raw, errors, decoded, expected))
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	820 else:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	821 self.assertRaises(UnicodeDecodeError,
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	822 raw.decode, 'cp65001', errors)
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	823
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	824 @unittest.skipUnless(VISTA_OR_LATER, 'require Windows Vista or later')
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	825 def test_lone_surrogates(self):
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	826 self.assertRaises(UnicodeEncodeError, "\ud800".encode, "cp65001")
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	827 self.assertRaises(UnicodeDecodeError, b"\xed\xa0\x80".decode, "cp65001")
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	828 self.assertEqual("[\uDC80]".encode("cp65001", "backslashreplace"),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	829 b'[\\udc80]')
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	830 self.assertEqual("[\uDC80]".encode("cp65001", "xmlcharrefreplace"),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	831 b'[&#56448;]')
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	832 self.assertEqual("[\uDC80]".encode("cp65001", "surrogateescape"),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	833 b'[\x80]')
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	834 self.assertEqual("[\uDC80]".encode("cp65001", "ignore"),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	835 b'[]')
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	836 self.assertEqual("[\uDC80]".encode("cp65001", "replace"),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	837 b'[?]')
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	838
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	839 @unittest.skipUnless(VISTA_OR_LATER, 'require Windows Vista or later')
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	840 def test_surrogatepass_handler(self):
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	841 self.assertEqual("abc\ud800def".encode("cp65001", "surrogatepass"),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	842 b"abc\xed\xa0\x80def")
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	843 self.assertEqual(b"abc\xed\xa0\x80def".decode("cp65001", "surrogatepass"),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	844 "abc\ud800def")
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	845 self.assertEqual("\U00010fff\uD800".encode("cp65001", "surrogatepass"),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	846 b"\xf0\x90\xbf\xbf\xed\xa0\x80")
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	847 self.assertEqual(b"\xf0\x90\xbf\xbf\xed\xa0\x80".decode("cp65001", "surrogatepass"),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	848 "\U00010fff\uD800")
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	849 self.assertTrue(codecs.lookup_error("surrogatepass"))
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	850
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	851
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	852
81383 11e7d0936fa2 #16918: test_codecs now works with unittest test discovery. Patch by Zachary Ware. Ezio Melotti <ezio.melotti@gmail.com> parents: 81382 diff changeset	853 class UTF7Test(ReadTest, unittest.TestCase):
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	854 encoding = "utf-7"
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	855
43933 9c4bf18e5d30 Merged revisions 59056-59076 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 43912 diff changeset	856 def test_partial(self):
9c4bf18e5d30 Merged revisions 59056-59076 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 43912 diff changeset	857 self.check_partial(
9c4bf18e5d30 Merged revisions 59056-59076 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 43912 diff changeset	858 "a+-b",
9c4bf18e5d30 Merged revisions 59056-59076 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 43912 diff changeset	859 [
9c4bf18e5d30 Merged revisions 59056-59076 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 43912 diff changeset	860 "a",
9c4bf18e5d30 Merged revisions 59056-59076 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 43912 diff changeset	861 "a",
9c4bf18e5d30 Merged revisions 59056-59076 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 43912 diff changeset	862 "a+",
9c4bf18e5d30 Merged revisions 59056-59076 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 43912 diff changeset	863 "a+-",
9c4bf18e5d30 Merged revisions 59056-59076 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 43912 diff changeset	864 "a+-b",
9c4bf18e5d30 Merged revisions 59056-59076 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 43912 diff changeset	865 ]
9c4bf18e5d30 Merged revisions 59056-59076 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 43912 diff changeset	866 )
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	867
86477 f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	868 def test_errors(self):
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	869 tests = [
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	870 (b'a\xffb', 'a\ufffdb'),
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	871 (b'a+IK', 'a\ufffd'),
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	872 (b'a+IK-b', 'a\ufffdb'),
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	873 (b'a+IK,b', 'a\ufffdb'),
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	874 (b'a+IKx', 'a\u20ac\ufffd'),
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	875 (b'a+IKx-b', 'a\u20ac\ufffdb'),
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	876 (b'a+IKwgr', 'a\u20ac\ufffd'),
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	877 (b'a+IKwgr-b', 'a\u20ac\ufffdb'),
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	878 (b'a+IKwgr,', 'a\u20ac\ufffd'),
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	879 (b'a+IKwgr,-b', 'a\u20ac\ufffd-b'),
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	880 (b'a+IKwgrB', 'a\u20ac\u20ac\ufffd'),
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	881 (b'a+IKwgrB-b', 'a\u20ac\u20ac\ufffdb'),
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	882 (b'a+/,+IKw-b', 'a\ufffd\u20acb'),
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	883 (b'a+//,+IKw-b', 'a\ufffd\u20acb'),
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	884 (b'a+///,+IKw-b', 'a\uffff\ufffd\u20acb'),
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	885 (b'a+////,+IKw-b', 'a\uffff\ufffd\u20acb'),
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	886 ]
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	887 for raw, expected in tests:
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	888 with self.subTest(raw=raw):
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	889 self.assertRaises(UnicodeDecodeError, codecs.utf_7_decode,
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	890 raw, 'strict', True)
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	891 self.assertEqual(raw.decode('utf-7', 'replace'), expected)
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	892
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	893 def test_nonbmp(self):
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	894 self.assertEqual('\U000104A0'.encode(self.encoding), b'+2AHcoA-')
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	895 self.assertEqual('\ud801\udca0'.encode(self.encoding), b'+2AHcoA-')
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	896 self.assertEqual(b'+2AHcoA-'.decode(self.encoding), '\U000104A0')
f471f2f05621 Issue #19279: UTF-7 decoder no more produces illegal strings. Serhiy Storchaka <storchaka@gmail.com> parents: 81819 diff changeset	897
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	898 test_lone_surrogates = None
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	899
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	900
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	901 class UTF16ExTest(unittest.TestCase):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	902
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	903 def test_errors(self):
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	904 self.assertRaises(UnicodeDecodeError, codecs.utf_16_ex_decode, b"\xff", "strict", 0, True)
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	905
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	906 def test_bad_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	907 self.assertRaises(TypeError, codecs.utf_16_ex_decode)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	908
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	909 class ReadBufferTest(unittest.TestCase):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	910
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	911 def test_array(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	912 import array
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	913 self.assertEqual(
42911 cb28e14ee808 Changes in anticipation of stricter str vs. bytes enforcement. Guido van Rossum <guido@python.org> parents: 42626 diff changeset	914 codecs.readbuffer_encode(array.array("b", b"spam")),
42124 4590129e4c2b Change readbuffer_encode() and charbuffer_encode() to Walter Dörwald <walter@livinglogic.de> parents: 42118 diff changeset	915 (b"spam", 4)
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	916 )
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	917
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	918 def test_empty(self):
42124 4590129e4c2b Change readbuffer_encode() and charbuffer_encode() to Walter Dörwald <walter@livinglogic.de> parents: 42118 diff changeset	919 self.assertEqual(codecs.readbuffer_encode(""), (b"", 0))
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	920
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	921 def test_bad_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	922 self.assertRaises(TypeError, codecs.readbuffer_encode)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	923 self.assertRaises(TypeError, codecs.readbuffer_encode, 42)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	924
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87111 diff changeset	925 class UTF8SigTest(UTF8Test, unittest.TestCase):
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	926 encoding = "utf-8-sig"
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	927
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	928 def test_partial(self):
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	929 self.check_partial(
81329 4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	930 "\ufeff\x00\xff\u07ff\u0800\uffff\U00010000",
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	931 [
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	932 "",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	933 "",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	934 "", # First BOM has been read and skipped
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	935 "",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	936 "",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	937 "\ufeff", # Second BOM has been read and emitted
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	938 "\ufeff\x00", # "\x00" read and emitted
42118 70a0831feda6 Patch by Ron Adam: Don't use u prefix in unicode error messages Walter Dörwald <walter@livinglogic.de> parents: 42064 diff changeset	939 "\ufeff\x00", # First byte of encoded "\xff" read
70a0831feda6 Patch by Ron Adam: Don't use u prefix in unicode error messages Walter Dörwald <walter@livinglogic.de> parents: 42064 diff changeset	940 "\ufeff\x00\xff", # Second byte of encoded "\xff" read
70a0831feda6 Patch by Ron Adam: Don't use u prefix in unicode error messages Walter Dörwald <walter@livinglogic.de> parents: 42064 diff changeset	941 "\ufeff\x00\xff", # First byte of encoded "\u07ff" read
70a0831feda6 Patch by Ron Adam: Don't use u prefix in unicode error messages Walter Dörwald <walter@livinglogic.de> parents: 42064 diff changeset	942 "\ufeff\x00\xff\u07ff", # Second byte of encoded "\u07ff" read
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	943 "\ufeff\x00\xff\u07ff",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	944 "\ufeff\x00\xff\u07ff",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	945 "\ufeff\x00\xff\u07ff\u0800",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	946 "\ufeff\x00\xff\u07ff\u0800",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	947 "\ufeff\x00\xff\u07ff\u0800",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	948 "\ufeff\x00\xff\u07ff\u0800\uffff",
81329 4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	949 "\ufeff\x00\xff\u07ff\u0800\uffff",
4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	950 "\ufeff\x00\xff\u07ff\u0800\uffff",
4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	951 "\ufeff\x00\xff\u07ff\u0800\uffff",
4677c5f6fcf7 Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Serhiy Storchaka <storchaka@gmail.com> parents: 80216 diff changeset	952 "\ufeff\x00\xff\u07ff\u0800\uffff\U00010000",
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	953 ]
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	954 )
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	955
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	956 def test_bug1601501(self):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	957 # SF bug #1601501: check that the codec works with a buffer
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	958 self.assertEqual(str(b"\xef\xbb\xbf", "utf-8-sig"), "")
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	959
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	960 def test_bom(self):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	961 d = codecs.getincrementaldecoder("utf-8-sig")()
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	962 s = "spam"
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	963 self.assertEqual(d.decode(s.encode("utf-8-sig")), s)
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	964
43912 4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	965 def test_stream_bom(self):
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	966 unistring = "ABC\u00A1\u2200XYZ"
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	967 bytestring = codecs.BOM_UTF8 + b"ABC\xC2\xA1\xE2\x88\x80XYZ"
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	968
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	969 reader = codecs.getreader("utf-8-sig")
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	970 for sizehint in [None] + list(range(1, 11)) + \
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	971 [64, 128, 256, 512, 1024]:
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	972 istream = reader(io.BytesIO(bytestring))
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	973 ostream = io.StringIO()
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	974 while 1:
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	975 if sizehint is not None:
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	976 data = istream.read(sizehint)
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	977 else:
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	978 data = istream.read()
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	979
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	980 if not data:
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	981 break
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	982 ostream.write(data)
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	983
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	984 got = ostream.getvalue()
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	985 self.assertEqual(got, unistring)
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	986
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	987 def test_stream_bare(self):
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	988 unistring = "ABC\u00A1\u2200XYZ"
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	989 bytestring = b"ABC\xC2\xA1\xE2\x88\x80XYZ"
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	990
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	991 reader = codecs.getreader("utf-8-sig")
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	992 for sizehint in [None] + list(range(1, 11)) + \
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	993 [64, 128, 256, 512, 1024]:
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	994 istream = reader(io.BytesIO(bytestring))
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	995 ostream = io.StringIO()
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	996 while 1:
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	997 if sizehint is not None:
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	998 data = istream.read(sizehint)
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	999 else:
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	1000 data = istream.read()
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	1001
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	1002 if not data:
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	1003 break
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	1004 ostream.write(data)
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	1005
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	1006 got = ostream.getvalue()
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	1007 self.assertEqual(got, unistring)
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	1008
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	1009 class EscapeDecodeTest(unittest.TestCase):
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	1010 def test_empty(self):
81814 946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1011 self.assertEqual(codecs.escape_decode(b""), (b"", 0))
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	1012
81738 22594c5060eb Issue #16975: Fix error handling bug in the escape-decode bytes decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81518 diff changeset	1013 def test_raw(self):
81814 946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1014 decode = codecs.escape_decode
81738 22594c5060eb Issue #16975: Fix error handling bug in the escape-decode bytes decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81518 diff changeset	1015 for b in range(256):
81814 946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1016 b = bytes([b])
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1017 if b != b'\\':
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1018 self.assertEqual(decode(b + b'0'), (b + b'0', 2))
81738 22594c5060eb Issue #16975: Fix error handling bug in the escape-decode bytes decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81518 diff changeset	1019
22594c5060eb Issue #16975: Fix error handling bug in the escape-decode bytes decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81518 diff changeset	1020 def test_escape(self):
81814 946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1021 decode = codecs.escape_decode
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1022 check = coding_checker(self, decode)
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1023 check(b"[\\\n]", b"[]")
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1024 check(br'[\"]', b'["]')
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1025 check(br"[\']", b"[']")
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1026 check(br"[\\]", br"[\]")
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1027 check(br"[\a]", b"[\x07]")
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1028 check(br"[\b]", b"[\x08]")
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1029 check(br"[\t]", b"[\x09]")
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1030 check(br"[\n]", b"[\x0a]")
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1031 check(br"[\v]", b"[\x0b]")
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1032 check(br"[\f]", b"[\x0c]")
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1033 check(br"[\r]", b"[\x0d]")
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1034 check(br"[\7]", b"[\x07]")
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1035 check(br"[\8]", br"[\8]")
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1036 check(br"[\78]", b"[\x078]")
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1037 check(br"[\41]", b"[!]")
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1038 check(br"[\418]", b"[!8]")
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1039 check(br"[\101]", b"[A]")
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1040 check(br"[\1010]", b"[A0]")
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1041 check(br"[\501]", b"[A]")
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1042 check(br"[\x41]", b"[A]")
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1043 check(br"[\X41]", br"[\X41]")
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1044 check(br"[\x410]", b"[A0]")
81738 22594c5060eb Issue #16975: Fix error handling bug in the escape-decode bytes decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81518 diff changeset	1045 for b in range(256):
22594c5060eb Issue #16975: Fix error handling bug in the escape-decode bytes decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81518 diff changeset	1046 if b not in b'\n"\'\\abtnvfr01234567x':
81814 946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1047 b = bytes([b])
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1048 check(b'\\' + b, b'\\' + b)
81738 22594c5060eb Issue #16975: Fix error handling bug in the escape-decode bytes decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81518 diff changeset	1049
22594c5060eb Issue #16975: Fix error handling bug in the escape-decode bytes decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81518 diff changeset	1050 def test_errors(self):
81814 946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1051 decode = codecs.escape_decode
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1052 self.assertRaises(ValueError, decode, br"\x")
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1053 self.assertRaises(ValueError, decode, br"[\x]")
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1054 self.assertEqual(decode(br"[\x]\x", "ignore"), (b"[]", 6))
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1055 self.assertEqual(decode(br"[\x]\x", "replace"), (b"[?]?", 6))
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1056 self.assertRaises(ValueError, decode, br"\x0")
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1057 self.assertRaises(ValueError, decode, br"[\x0]")
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1058 self.assertEqual(decode(br"[\x0]\x0", "ignore"), (b"[]", 8))
946df0c26370 Clean up escape-decode decoder tests. Serhiy Storchaka <storchaka@gmail.com> parents: 81804 diff changeset	1059 self.assertEqual(decode(br"[\x0]\x0", "replace"), (b"[?]?", 8))
81738 22594c5060eb Issue #16975: Fix error handling bug in the escape-decode bytes decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81518 diff changeset	1060
27680 b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	1061 class RecodingTest(unittest.TestCase):
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	1062 def test_recoding(self):
41757 13a6c6bef609 Make test_codecs work. The CJK codecs now use bytes instead of str8 for Guido van Rossum <guido@python.org> parents: 41750 diff changeset	1063 f = io.BytesIO()
27680 b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	1064 f2 = codecs.EncodedFile(f, "unicode_internal", "utf-8")
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1065 f2.write("a")
27680 b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	1066 f2.close()
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	1067 # Python used to crash on this at exit because of a refcount
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	1068 # bug in _codecsmodule.c
20013 e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	1069
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1070 # From RFC 3492
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1071 punycode_testcases = [
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1072 # A Arabic (Egyptian):
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1073 ("\u0644\u064A\u0647\u0645\u0627\u0628\u062A\u0643\u0644"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1074 "\u0645\u0648\u0634\u0639\u0631\u0628\u064A\u061F",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1075 b"egbpdaj6bu4bxfgehfvwxn"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1076 # B Chinese (simplified):
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1077 ("\u4ED6\u4EEC\u4E3A\u4EC0\u4E48\u4E0D\u8BF4\u4E2D\u6587",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1078 b"ihqwcrb4cv8a8dqg056pqjye"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1079 # C Chinese (traditional):
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1080 ("\u4ED6\u5011\u7232\u4EC0\u9EBD\u4E0D\u8AAA\u4E2D\u6587",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1081 b"ihqwctvzc91f659drss3x8bo0yb"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1082 # D Czech: Pro<ccaron>prost<ecaron>nemluv<iacute><ccaron>esky
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1083 ("\u0050\u0072\u006F\u010D\u0070\u0072\u006F\u0073\u0074"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1084 "\u011B\u006E\u0065\u006D\u006C\u0075\u0076\u00ED\u010D"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1085 "\u0065\u0073\u006B\u0079",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1086 b"Proprostnemluvesky-uyb24dma41a"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1087 # E Hebrew:
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1088 ("\u05DC\u05DE\u05D4\u05D4\u05DD\u05E4\u05E9\u05D5\u05D8"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1089 "\u05DC\u05D0\u05DE\u05D3\u05D1\u05E8\u05D9\u05DD\u05E2"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1090 "\u05D1\u05E8\u05D9\u05EA",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1091 b"4dbcagdahymbxekheh6e0a7fei0b"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1092 # F Hindi (Devanagari):
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1093 ("\u092F\u0939\u0932\u094B\u0917\u0939\u093F\u0928\u094D"
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1094 "\u0926\u0940\u0915\u094D\u092F\u094B\u0902\u0928\u0939"
98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1095 "\u0940\u0902\u092C\u094B\u0932\u0938\u0915\u0924\u0947"
98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1096 "\u0939\u0948\u0902",
98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1097 b"i1baa7eci9glrd9b2ae1bj0hfcgg6iyaf8o0a1dig0cd"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1098
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1099 #(G) Japanese (kanji and hiragana):
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1100 ("\u306A\u305C\u307F\u3093\u306A\u65E5\u672C\u8A9E\u3092"
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1101 "\u8A71\u3057\u3066\u304F\u308C\u306A\u3044\u306E\u304B",
98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1102 b"n8jok5ay5dzabd5bym9f0cm5685rrjetr6pdxa"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1103
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1104 # (H) Korean (Hangul syllables):
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1105 ("\uC138\uACC4\uC758\uBAA8\uB4E0\uC0AC\uB78C\uB4E4\uC774"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1106 "\uD55C\uAD6D\uC5B4\uB97C\uC774\uD574\uD55C\uB2E4\uBA74"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1107 "\uC5BC\uB9C8\uB098\uC88B\uC744\uAE4C",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1108 b"989aomsvi5e83db1d2a355cv1e0vak1dwrv93d5xbh15a0dt30a5j"
98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1109 b"psd879ccm6fea98c"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1110
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1111 # (I) Russian (Cyrillic):
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1112 ("\u043F\u043E\u0447\u0435\u043C\u0443\u0436\u0435\u043E"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1113 "\u043D\u0438\u043D\u0435\u0433\u043E\u0432\u043E\u0440"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1114 "\u044F\u0442\u043F\u043E\u0440\u0443\u0441\u0441\u043A"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1115 "\u0438",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1116 b"b1abfaaepdrnnbgefbaDotcwatmq2g4l"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1117
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1118 # (J) Spanish: Porqu<eacute>nopuedensimplementehablarenEspa<ntilde>ol
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1119 ("\u0050\u006F\u0072\u0071\u0075\u00E9\u006E\u006F\u0070"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1120 "\u0075\u0065\u0064\u0065\u006E\u0073\u0069\u006D\u0070"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1121 "\u006C\u0065\u006D\u0065\u006E\u0074\u0065\u0068\u0061"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1122 "\u0062\u006C\u0061\u0072\u0065\u006E\u0045\u0073\u0070"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1123 "\u0061\u00F1\u006F\u006C",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1124 b"PorqunopuedensimplementehablarenEspaol-fmd56a"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1125
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1126 # (K) Vietnamese:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1127 # T<adotbelow>isaoh<odotbelow>kh<ocirc>ngth<ecirchookabove>ch\
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1128 # <ihookabove>n<oacute>iti<ecircacute>ngVi<ecircdotbelow>t
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1129 ("\u0054\u1EA1\u0069\u0073\u0061\u006F\u0068\u1ECD\u006B"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1130 "\u0068\u00F4\u006E\u0067\u0074\u0068\u1EC3\u0063\u0068"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1131 "\u1EC9\u006E\u00F3\u0069\u0074\u0069\u1EBF\u006E\u0067"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1132 "\u0056\u0069\u1EC7\u0074",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1133 b"TisaohkhngthchnitingVit-kjcr8268qyxafd2f1b9g"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1134
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1135 #(L) 3<nen>B<gumi><kinpachi><sensei>
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1136 ("\u0033\u5E74\u0042\u7D44\u91D1\u516B\u5148\u751F",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1137 b"3B-ww4c5e180e575a65lsy2b"),
28774 c9bee11eaa8b Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 28678 diff changeset	1138
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1139 # (M) <amuro><namie>-with-SUPER-MONKEYS
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1140 ("\u5B89\u5BA4\u5948\u7F8E\u6075\u002D\u0077\u0069\u0074"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1141 "\u0068\u002D\u0053\u0055\u0050\u0045\u0052\u002D\u004D"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1142 "\u004F\u004E\u004B\u0045\u0059\u0053",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1143 b"-with-SUPER-MONKEYS-pc58ag80a8qai00g7n9n"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1144
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1145 # (N) Hello-Another-Way-<sorezore><no><basho>
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1146 ("\u0048\u0065\u006C\u006C\u006F\u002D\u0041\u006E\u006F"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1147 "\u0074\u0068\u0065\u0072\u002D\u0057\u0061\u0079\u002D"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1148 "\u305D\u308C\u305E\u308C\u306E\u5834\u6240",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1149 b"Hello-Another-Way--fc4qua05auwb3674vfr0b"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1150
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1151 # (O) <hitotsu><yane><no><shita>2
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1152 ("\u3072\u3068\u3064\u5C4B\u6839\u306E\u4E0B\u0032",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1153 b"2-u9tlzr9756bt3uc0v"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1154
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1155 # (P) Maji<de>Koi<suru>5<byou><mae>
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1156 ("\u004D\u0061\u006A\u0069\u3067\u004B\u006F\u0069\u3059"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1157 "\u308B\u0035\u79D2\u524D",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1158 b"MajiKoi5-783gue6qz075azm5e"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1159
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1160 # (Q) <pafii>de<runba>
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1161 ("\u30D1\u30D5\u30A3\u30FC\u0064\u0065\u30EB\u30F3\u30D0",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1162 b"de-jg4avhby1noc0d"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1163
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1164 # (R) <sono><supiido><de>
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1165 ("\u305D\u306E\u30B9\u30D4\u30FC\u30C9\u3067",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1166 b"d9juau41awczczp"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1167
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1168 # (S) -> $1.00 <-
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1169 ("\u002D\u003E\u0020\u0024\u0031\u002E\u0030\u0030\u0020"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1170 "\u003C\u002D",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1171 b"-> $1.00 <--")
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1172 ]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1173
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1174 for i in punycode_testcases:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1175 if len(i)!=2:
40775 4cdf21a0dbbc Fix most trivially-findable print statements. Guido van Rossum <guido@python.org> parents: 40587 diff changeset	1176 print(repr(i))
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1177
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1178 class PunycodeTest(unittest.TestCase):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1179 def test_encode(self):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1180 for uni, puny in punycode_testcases:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1181 # Need to convert both strings to lower case, since
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1182 # some of the extended encodings use upper case, but our
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1183 # code produces only lower case. Converting just puny to
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1184 # lower is also insufficient, since some of the input characters
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1185 # are upper case.
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1186 self.assertEqual(
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1187 str(uni.encode("punycode"), "ascii").lower(),
98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1188 str(puny, "ascii").lower()
98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1189 )
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1190
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1191 def test_decode(self):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1192 for uni, puny in punycode_testcases:
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1193 self.assertEqual(uni, puny.decode("punycode"))
42985 f4851dc0df04 Fix the one failing test (can't decode twice). Guido van Rossum <guido@python.org> parents: 42911 diff changeset	1194 puny = puny.decode("ascii").encode("ascii")
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1195 self.assertEqual(uni, puny.decode("punycode"))
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1196
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	1197 class UnicodeInternalTest(unittest.TestCase):
72534 85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1198 @unittest.skipUnless(SIZEOF_WCHAR_T == 4, 'specific to 32-bit wchar_t')
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	1199 def test_bug1251300(self):
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	1200 # Decoding with unicode_internal used to not correctly handle "code
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	1201 # points" above 0x10ffff on UCS-4 builds.
72534 85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1202 ok = [
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1203 (b"\x00\x10\xff\xff", "\U0010ffff"),
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1204 (b"\x00\x00\x01\x01", "\U00000101"),
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1205 (b"", ""),
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1206 ]
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1207 not_ok = [
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1208 b"\x7f\xff\xff\xff",
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1209 b"\x80\x00\x00\x00",
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1210 b"\x81\x00\x00\x00",
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1211 b"\x00",
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1212 b"\x00\x00\x00\x00\x00",
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1213 ]
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1214 for internal, uni in ok:
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1215 if sys.byteorder == "little":
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1216 internal = bytes(reversed(internal))
73588 4f534cd40f54 #13406: silence deprecation warnings in test_codecs. Ezio Melotti <ezio.melotti@gmail.com> parents: 73579 diff changeset	1217 with support.check_warnings():
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1218 self.assertEqual(uni, internal.decode("unicode_internal"))
72534 85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1219 for internal in not_ok:
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1220 if sys.byteorder == "little":
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1221 internal = bytes(reversed(internal))
73589 028b61c4f926 #13406: correct the error message in check_warnings too. Ezio Melotti <ezio.melotti@gmail.com> parents: 73588 diff changeset	1222 with support.check_warnings(('unicode_internal codec has been '
73588 4f534cd40f54 #13406: silence deprecation warnings in test_codecs. Ezio Melotti <ezio.melotti@gmail.com> parents: 73579 diff changeset	1223 'deprecated', DeprecationWarning)):
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	1224 self.assertRaises(UnicodeDecodeError, internal.decode,
73588 4f534cd40f54 #13406: silence deprecation warnings in test_codecs. Ezio Melotti <ezio.melotti@gmail.com> parents: 73579 diff changeset	1225 "unicode_internal")
73906 bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	1226 if sys.byteorder == "little":
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	1227 invalid = b"\x00\x00\x11\x00"
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	1228 else:
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	1229 invalid = b"\x00\x11\x00\x00"
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	1230 with support.check_warnings():
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	1231 self.assertRaises(UnicodeDecodeError,
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	1232 invalid.decode, "unicode_internal")
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	1233 with support.check_warnings():
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	1234 self.assertEqual(invalid.decode("unicode_internal", "replace"),
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	1235 '\ufffd')
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	1236
72534 85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1237 @unittest.skipUnless(SIZEOF_WCHAR_T == 4, 'specific to 32-bit wchar_t')
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	1238 def test_decode_error_attributes(self):
72534 85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1239 try:
73589 028b61c4f926 #13406: correct the error message in check_warnings too. Ezio Melotti <ezio.melotti@gmail.com> parents: 73588 diff changeset	1240 with support.check_warnings(('unicode_internal codec has been '
73588 4f534cd40f54 #13406: silence deprecation warnings in test_codecs. Ezio Melotti <ezio.melotti@gmail.com> parents: 73579 diff changeset	1241 'deprecated', DeprecationWarning)):
41971 a3261cde3a37 Fix tests for unicode-internal codec. Walter Dörwald <walter@livinglogic.de> parents: 41757 diff changeset	1242 b"\x00\x00\x00\x00\x00\x11\x11\x00".decode("unicode_internal")
72534 85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1243 except UnicodeDecodeError as ex:
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1244 self.assertEqual("unicode_internal", ex.encoding)
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1245 self.assertEqual(b"\x00\x00\x00\x00\x00\x11\x11\x00", ex.object)
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1246 self.assertEqual(4, ex.start)
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1247 self.assertEqual(8, ex.end)
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1248 else:
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1249 self.fail()
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	1250
72534 85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1251 @unittest.skipUnless(SIZEOF_WCHAR_T == 4, 'specific to 32-bit wchar_t')
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	1252 def test_decode_callback(self):
72534 85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1253 codecs.register_error("UnicodeInternalTest", codecs.ignore_errors)
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1254 decoder = codecs.getdecoder("unicode_internal")
73589 028b61c4f926 #13406: correct the error message in check_warnings too. Ezio Melotti <ezio.melotti@gmail.com> parents: 73588 diff changeset	1255 with support.check_warnings(('unicode_internal codec has been '
73588 4f534cd40f54 #13406: silence deprecation warnings in test_codecs. Ezio Melotti <ezio.melotti@gmail.com> parents: 73579 diff changeset	1256 'deprecated', DeprecationWarning)):
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43577 diff changeset	1257 ab = "ab".encode("unicode_internal").decode()
43577 ace45b361e87 Patch# 1258 by Christian Heimes: kill basestring. Guido van Rossum <guido@python.org> parents: 42985 diff changeset	1258 ignored = decoder(bytes("%s\x22\x22\x22\x22%s" % (ab[:4], ab[4:]),
ace45b361e87 Patch# 1258 by Christian Heimes: kill basestring. Guido van Rossum <guido@python.org> parents: 42985 diff changeset	1259 "ascii"),
ace45b361e87 Patch# 1258 by Christian Heimes: kill basestring. Guido van Rossum <guido@python.org> parents: 42985 diff changeset	1260 "UnicodeInternalTest")
72534 85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1261 self.assertEqual(("ab", 12), ignored)
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	1262
53859 63b23bd3904d Merged revisions 72404-72406 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 53767 diff changeset	1263 def test_encode_length(self):
73600 fe2be7d35660 #13406: fix more deprecation warnings and move the deprecation of unicode-internal earlier in the code. Ezio Melotti <ezio.melotti@gmail.com> parents: 73589 diff changeset	1264 with support.check_warnings(('unicode_internal codec has been '
fe2be7d35660 #13406: fix more deprecation warnings and move the deprecation of unicode-internal earlier in the code. Ezio Melotti <ezio.melotti@gmail.com> parents: 73589 diff changeset	1265 'deprecated', DeprecationWarning)):
73579 ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	1266 # Issue 3739
ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	1267 encoder = codecs.getencoder("unicode_internal")
ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	1268 self.assertEqual(encoder("a")[1], 1)
ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	1269 self.assertEqual(encoder("\xe9\u0142")[1], 2)
53859 63b23bd3904d Merged revisions 72404-72406 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 53767 diff changeset	1270
73579 ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	1271 self.assertEqual(codecs.escape_encode(br'\x00')[1], 4)
62215 f33e464a9a60 Merged revisions 79780 via svnmerge from Philip Jenvey <pjenvey@underboss.org> parents: 61863 diff changeset	1272
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1273 # From http://www.gnu.org/software/libidn/draft-josefsson-idn-test-vectors.html
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1274 nameprep_tests = [
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1275 # 3.1 Map to nothing.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1276 (b'foo\xc2\xad\xcd\x8f\xe1\xa0\x86\xe1\xa0\x8bbar'
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1277 b'\xe2\x80\x8b\xe2\x81\xa0baz\xef\xb8\x80\xef\xb8\x88\xef'
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1278 b'\xb8\x8f\xef\xbb\xbf',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1279 b'foobarbaz'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1280 # 3.2 Case folding ASCII U+0043 U+0041 U+0046 U+0045.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1281 (b'CAFE',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1282 b'cafe'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1283 # 3.3 Case folding 8bit U+00DF (german sharp s).
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1284 # The original test case is bogus; it says \xc3\xdf
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1285 (b'\xc3\x9f',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1286 b'ss'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1287 # 3.4 Case folding U+0130 (turkish capital I with dot).
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1288 (b'\xc4\xb0',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1289 b'i\xcc\x87'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1290 # 3.5 Case folding multibyte U+0143 U+037A.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1291 (b'\xc5\x83\xcd\xba',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1292 b'\xc5\x84 \xce\xb9'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1293 # 3.6 Case folding U+2121 U+33C6 U+1D7BB.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1294 # XXX: skip this as it fails in UCS-2 mode
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1295 #('\xe2\x84\xa1\xe3\x8f\x86\xf0\x9d\x9e\xbb',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1296 # 'telc\xe2\x88\x95kg\xcf\x83'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1297 (None, None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1298 # 3.7 Normalization of U+006a U+030c U+00A0 U+00AA.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1299 (b'j\xcc\x8c\xc2\xa0\xc2\xaa',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1300 b'\xc7\xb0 a'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1301 # 3.8 Case folding U+1FB7 and normalization.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1302 (b'\xe1\xbe\xb7',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1303 b'\xe1\xbe\xb6\xce\xb9'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1304 # 3.9 Self-reverting case folding U+01F0 and normalization.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1305 # The original test case is bogus, it says `\xc7\xf0'
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1306 (b'\xc7\xb0',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1307 b'\xc7\xb0'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1308 # 3.10 Self-reverting case folding U+0390 and normalization.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1309 (b'\xce\x90',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1310 b'\xce\x90'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1311 # 3.11 Self-reverting case folding U+03B0 and normalization.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1312 (b'\xce\xb0',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1313 b'\xce\xb0'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1314 # 3.12 Self-reverting case folding U+1E96 and normalization.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1315 (b'\xe1\xba\x96',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1316 b'\xe1\xba\x96'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1317 # 3.13 Self-reverting case folding U+1F56 and normalization.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1318 (b'\xe1\xbd\x96',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1319 b'\xe1\xbd\x96'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1320 # 3.14 ASCII space character U+0020.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1321 (b' ',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1322 b' '),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1323 # 3.15 Non-ASCII 8bit space character U+00A0.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1324 (b'\xc2\xa0',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1325 b' '),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1326 # 3.16 Non-ASCII multibyte space character U+1680.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1327 (b'\xe1\x9a\x80',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1328 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1329 # 3.17 Non-ASCII multibyte space character U+2000.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1330 (b'\xe2\x80\x80',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1331 b' '),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1332 # 3.18 Zero Width Space U+200b.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1333 (b'\xe2\x80\x8b',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1334 b''),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1335 # 3.19 Non-ASCII multibyte space character U+3000.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1336 (b'\xe3\x80\x80',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1337 b' '),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1338 # 3.20 ASCII control characters U+0010 U+007F.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1339 (b'\x10\x7f',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1340 b'\x10\x7f'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1341 # 3.21 Non-ASCII 8bit control character U+0085.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1342 (b'\xc2\x85',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1343 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1344 # 3.22 Non-ASCII multibyte control character U+180E.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1345 (b'\xe1\xa0\x8e',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1346 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1347 # 3.23 Zero Width No-Break Space U+FEFF.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1348 (b'\xef\xbb\xbf',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1349 b''),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1350 # 3.24 Non-ASCII control character U+1D175.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1351 (b'\xf0\x9d\x85\xb5',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1352 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1353 # 3.25 Plane 0 private use character U+F123.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1354 (b'\xef\x84\xa3',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1355 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1356 # 3.26 Plane 15 private use character U+F1234.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1357 (b'\xf3\xb1\x88\xb4',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1358 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1359 # 3.27 Plane 16 private use character U+10F234.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1360 (b'\xf4\x8f\x88\xb4',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1361 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1362 # 3.28 Non-character code point U+8FFFE.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1363 (b'\xf2\x8f\xbf\xbe',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1364 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1365 # 3.29 Non-character code point U+10FFFF.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1366 (b'\xf4\x8f\xbf\xbf',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1367 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1368 # 3.30 Surrogate code U+DF42.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1369 (b'\xed\xbd\x82',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1370 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1371 # 3.31 Non-plain text character U+FFFD.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1372 (b'\xef\xbf\xbd',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1373 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1374 # 3.32 Ideographic description character U+2FF5.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1375 (b'\xe2\xbf\xb5',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1376 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1377 # 3.33 Display property character U+0341.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1378 (b'\xcd\x81',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1379 b'\xcc\x81'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1380 # 3.34 Left-to-right mark U+200E.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1381 (b'\xe2\x80\x8e',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1382 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1383 # 3.35 Deprecated U+202A.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1384 (b'\xe2\x80\xaa',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1385 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1386 # 3.36 Language tagging character U+E0001.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1387 (b'\xf3\xa0\x80\x81',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1388 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1389 # 3.37 Language tagging character U+E0042.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1390 (b'\xf3\xa0\x81\x82',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1391 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1392 # 3.38 Bidi: RandALCat character U+05BE and LCat characters.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1393 (b'foo\xd6\xbebar',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1394 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1395 # 3.39 Bidi: RandALCat character U+FD50 and LCat characters.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1396 (b'foo\xef\xb5\x90bar',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1397 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1398 # 3.40 Bidi: RandALCat character U+FB38 and LCat characters.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1399 (b'foo\xef\xb9\xb6bar',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1400 b'foo \xd9\x8ebar'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1401 # 3.41 Bidi: RandALCat without trailing RandALCat U+0627 U+0031.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1402 (b'\xd8\xa71',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1403 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1404 # 3.42 Bidi: RandALCat character U+0627 U+0031 U+0628.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1405 (b'\xd8\xa71\xd8\xa8',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1406 b'\xd8\xa71\xd8\xa8'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1407 # 3.43 Unassigned code point U+E0002.
28678 31cb8e0260db Skip nameprep test 3.43, as we do allow unassigned characters. The test Martin v. Löwis <martin@v.loewis.de> parents: 28671 diff changeset	1408 # Skip this test as we allow unassigned
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1409 #(b'\xf3\xa0\x80\x82',
28678 31cb8e0260db Skip nameprep test 3.43, as we do allow unassigned characters. The test Martin v. Löwis <martin@v.loewis.de> parents: 28671 diff changeset	1410 # None),
31cb8e0260db Skip nameprep test 3.43, as we do allow unassigned characters. The test Martin v. Löwis <martin@v.loewis.de> parents: 28671 diff changeset	1411 (None, None),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1412 # 3.44 Larger test (shrinking).
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1413 # Original test case reads \xc3\xdf
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1414 (b'X\xc2\xad\xc3\x9f\xc4\xb0\xe2\x84\xa1j\xcc\x8c\xc2\xa0\xc2'
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1415 b'\xaa\xce\xb0\xe2\x80\x80',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1416 b'xssi\xcc\x87tel\xc7\xb0 a\xce\xb0 '),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1417 # 3.45 Larger test (expanding).
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1418 # Original test case reads \xc3\x9f
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1419 (b'X\xc3\x9f\xe3\x8c\x96\xc4\xb0\xe2\x84\xa1\xe2\x92\x9f\xe3\x8c'
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1420 b'\x80',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1421 b'xss\xe3\x82\xad\xe3\x83\xad\xe3\x83\xa1\xe3\x83\xbc\xe3'
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1422 b'\x83\x88\xe3\x83\xabi\xcc\x87tel\x28d\x29\xe3\x82'
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1423 b'\xa2\xe3\x83\x91\xe3\x83\xbc\xe3\x83\x88')
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1424 ]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1425
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1426
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1427 class NameprepTest(unittest.TestCase):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1428 def test_nameprep(self):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1429 from encodings.idna import nameprep
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1430 for pos, (orig, prepped) in enumerate(nameprep_tests):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1431 if orig is None:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1432 # Skipped
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1433 continue
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1434 # The Unicode strings are given in UTF-8
53965 fbcdae3e0ca1 Rename the surrogates error handler to surrogatepass. Martin v. Löwis <martin@v.loewis.de> parents: 53859 diff changeset	1435 orig = str(orig, "utf-8", "surrogatepass")
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1436 if prepped is None:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1437 # Input contains prohibited characters
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1438 self.assertRaises(UnicodeError, nameprep, orig)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1439 else:
53965 fbcdae3e0ca1 Rename the surrogates error handler to surrogatepass. Martin v. Löwis <martin@v.loewis.de> parents: 53859 diff changeset	1440 prepped = str(prepped, "utf-8", "surrogatepass")
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1441 try:
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1442 self.assertEqual(nameprep(orig), prepped)
40587 6b18a095c2e7 SF patch 1631942 by Collin Winter: Guido van Rossum <guido@python.org> parents: 40450 diff changeset	1443 except Exception as e:
47201 9c930a32cebb #2621 rename test.test_support to test.support Benjamin Peterson <benjamin@python.org> parents: 43933 diff changeset	1444 raise support.TestFailed("Test 3.%d: %s" % (pos+1, str(e)))
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1445
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1446 class IDNACodecTest(unittest.TestCase):
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1447 def test_builtin_decode(self):
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1448 self.assertEqual(str(b"python.org", "idna"), "python.org")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1449 self.assertEqual(str(b"python.org.", "idna"), "python.org.")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1450 self.assertEqual(str(b"xn--pythn-mua.org", "idna"), "pyth\xf6n.org")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1451 self.assertEqual(str(b"xn--pythn-mua.org.", "idna"), "pyth\xf6n.org.")
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1452
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1453 def test_builtin_encode(self):
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1454 self.assertEqual("python.org".encode("idna"), b"python.org")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1455 self.assertEqual("python.org.".encode("idna"), b"python.org.")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1456 self.assertEqual("pyth\xf6n.org".encode("idna"), b"xn--pythn-mua.org")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1457 self.assertEqual("pyth\xf6n.org.".encode("idna"), b"xn--pythn-mua.org.")
31795 bf37429f6d9f Add test case for unicode(somestring, "idna"). Martin v. Löwis <martin@v.loewis.de> parents: 28917 diff changeset	1458
35435 9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	1459 def test_stream(self):
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	1460 r = codecs.getreader("idna")(io.BytesIO(b"abc"))
35435 9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	1461 r.read(3)
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1462 self.assertEqual(r.read(), "")
35435 9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	1463
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1464 def test_incremental_decode(self):
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1465 self.assertEqual(
42911 cb28e14ee808 Changes in anticipation of stricter str vs. bytes enforcement. Guido van Rossum <guido@python.org> parents: 42626 diff changeset	1466 "".join(codecs.iterdecode((bytes([c]) for c in b"python.org"), "idna")),
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1467 "python.org"
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1468 )
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1469 self.assertEqual(
42911 cb28e14ee808 Changes in anticipation of stricter str vs. bytes enforcement. Guido van Rossum <guido@python.org> parents: 42626 diff changeset	1470 "".join(codecs.iterdecode((bytes([c]) for c in b"python.org."), "idna")),
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1471 "python.org."
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1472 )
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1473 self.assertEqual(
42911 cb28e14ee808 Changes in anticipation of stricter str vs. bytes enforcement. Guido van Rossum <guido@python.org> parents: 42626 diff changeset	1474 "".join(codecs.iterdecode((bytes([c]) for c in b"xn--pythn-mua.org."), "idna")),
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1475 "pyth\xf6n.org."
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1476 )
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1477 self.assertEqual(
42911 cb28e14ee808 Changes in anticipation of stricter str vs. bytes enforcement. Guido van Rossum <guido@python.org> parents: 42626 diff changeset	1478 "".join(codecs.iterdecode((bytes([c]) for c in b"xn--pythn-mua.org."), "idna")),
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1479 "pyth\xf6n.org."
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1480 )
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1481
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1482 decoder = codecs.getincrementaldecoder("idna")()
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1483 self.assertEqual(decoder.decode(b"xn--xam", ), "")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1484 self.assertEqual(decoder.decode(b"ple-9ta.o", ), "\xe4xample.")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1485 self.assertEqual(decoder.decode(b"rg"), "")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1486 self.assertEqual(decoder.decode(b"", True), "org")
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1487
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1488 decoder.reset()
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1489 self.assertEqual(decoder.decode(b"xn--xam", ), "")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1490 self.assertEqual(decoder.decode(b"ple-9ta.o", ), "\xe4xample.")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1491 self.assertEqual(decoder.decode(b"rg."), "org.")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1492 self.assertEqual(decoder.decode(b"", True), "")
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1493
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1494 def test_incremental_encode(self):
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1495 self.assertEqual(
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1496 b"".join(codecs.iterencode("python.org", "idna")),
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1497 b"python.org"
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1498 )
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1499 self.assertEqual(
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1500 b"".join(codecs.iterencode("python.org.", "idna")),
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1501 b"python.org."
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1502 )
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1503 self.assertEqual(
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1504 b"".join(codecs.iterencode("pyth\xf6n.org.", "idna")),
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1505 b"xn--pythn-mua.org."
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1506 )
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1507 self.assertEqual(
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1508 b"".join(codecs.iterencode("pyth\xf6n.org.", "idna")),
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1509 b"xn--pythn-mua.org."
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1510 )
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1511
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1512 encoder = codecs.getincrementalencoder("idna")()
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1513 self.assertEqual(encoder.encode("\xe4x"), b"")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1514 self.assertEqual(encoder.encode("ample.org"), b"xn--xample-9ta.")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1515 self.assertEqual(encoder.encode("", True), b"org")
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1516
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1517 encoder.reset()
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1518 self.assertEqual(encoder.encode("\xe4x"), b"")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1519 self.assertEqual(encoder.encode("ample.org."), b"xn--xample-9ta.org.")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1520 self.assertEqual(encoder.encode("", True), b"")
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1521
32572 8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1522 class CodecsModuleTest(unittest.TestCase):
8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1523
8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1524 def test_decode(self):
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1525 self.assertEqual(codecs.decode(b'\xe4\xf6\xfc', 'latin-1'),
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1526 '\xe4\xf6\xfc')
33855 1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1527 self.assertRaises(TypeError, codecs.decode)
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1528 self.assertEqual(codecs.decode(b'abc'), 'abc')
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1529 self.assertRaises(UnicodeDecodeError, codecs.decode, b'\xff', 'ascii')
33855 1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1530
32572 8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1531 def test_encode(self):
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1532 self.assertEqual(codecs.encode('\xe4\xf6\xfc', 'latin-1'),
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1533 b'\xe4\xf6\xfc')
33855 1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1534 self.assertRaises(TypeError, codecs.encode)
35889 e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1535 self.assertRaises(LookupError, codecs.encode, "foo", "__spam__")
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1536 self.assertEqual(codecs.encode('abc'), b'abc')
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1537 self.assertRaises(UnicodeEncodeError, codecs.encode, '\xffff', 'ascii')
33855 1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1538
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1539 def test_register(self):
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1540 self.assertRaises(TypeError, codecs.register)
35889 e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1541 self.assertRaises(TypeError, codecs.register, 42)
33855 1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1542
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1543 def test_lookup(self):
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1544 self.assertRaises(TypeError, codecs.lookup)
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1545 self.assertRaises(LookupError, codecs.lookup, "__spam__")
35889 e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1546 self.assertRaises(LookupError, codecs.lookup, " ")
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1547
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1548 def test_getencoder(self):
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1549 self.assertRaises(TypeError, codecs.getencoder)
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1550 self.assertRaises(LookupError, codecs.getencoder, "__spam__")
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1551
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1552 def test_getdecoder(self):
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1553 self.assertRaises(TypeError, codecs.getdecoder)
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1554 self.assertRaises(LookupError, codecs.getdecoder, "__spam__")
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1555
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1556 def test_getreader(self):
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1557 self.assertRaises(TypeError, codecs.getreader)
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1558 self.assertRaises(LookupError, codecs.getreader, "__spam__")
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1559
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1560 def test_getwriter(self):
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1561 self.assertRaises(TypeError, codecs.getwriter)
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1562 self.assertRaises(LookupError, codecs.getwriter, "__spam__")
32572 8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1563
71489 92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70293 diff changeset	1564 def test_lookup_issue1813(self):
92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70293 diff changeset	1565 # Issue #1813: under Turkish locales, lookup of some codecs failed
92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70293 diff changeset	1566 # because 'I' is lowercased as "ı" (dotless i)
71511 7063aa2fa127 Backport 0398f07d4827 (fix for weird buildbot failures) Antoine Pitrou <solipsis@pitrou.net> parents: 71489 diff changeset	1567 oldlocale = locale.setlocale(locale.LC_CTYPE)
71489 92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70293 diff changeset	1568 self.addCleanup(locale.setlocale, locale.LC_CTYPE, oldlocale)
92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70293 diff changeset	1569 try:
92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70293 diff changeset	1570 locale.setlocale(locale.LC_CTYPE, 'tr_TR')
92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70293 diff changeset	1571 except locale.Error:
92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70293 diff changeset	1572 # Unsupported locale on this system
92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70293 diff changeset	1573 self.skipTest('test needs Turkish locale')
92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70293 diff changeset	1574 c = codecs.lookup('ASCII')
92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70293 diff changeset	1575 self.assertEqual(c.name, 'ascii')
92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70293 diff changeset	1576
33782 a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1577 class StreamReaderTest(unittest.TestCase):
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1578
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1579 def setUp(self):
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1580 self.reader = codecs.getreader('utf-8')
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	1581 self.stream = io.BytesIO(b'\xed\x95\x9c\n\xea\xb8\x80')
33782 a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1582
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1583 def test_readlines(self):
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1584 f = self.reader(self.stream)
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1585 self.assertEqual(f.readlines(), ['\ud55c\n', '\uae00'])
33782 a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1586
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1587 class EncodedFileTest(unittest.TestCase):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1588
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1589 def test_basic(self):
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	1590 f = io.BytesIO(b'\xed\x95\x9c\n\xea\xb8\x80')
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1591 ef = codecs.EncodedFile(f, 'utf-16-le', 'utf-8')
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1592 self.assertEqual(ef.read(), b'\\\xd5\n\x00\x00\xae')
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1593
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	1594 f = io.BytesIO()
68142 be6c38d1817b Normalize the encoding names for Latin-1 and UTF-8 to Marc-André Lemburg <mal@egenix.com> parents: 66837 diff changeset	1595 ef = codecs.EncodedFile(f, 'utf-8', 'latin-1')
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1596 ef.write(b'\xc3\xbc')
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1597 self.assertEqual(f.getvalue(), b'\xfc')
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1598
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1599 all_unicode_encodings = [
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1600 "ascii",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1601 "big5",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1602 "big5hkscs",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1603 "charmap",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1604 "cp037",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1605 "cp1006",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1606 "cp1026",
87430 d0fd68ef1aa9 Issue #19668: Added support for the cp1125 encoding. Serhiy Storchaka <storchaka@gmail.com> parents: 87388 diff changeset	1607 "cp1125",
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1608 "cp1140",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1609 "cp1250",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1610 "cp1251",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1611 "cp1252",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1612 "cp1253",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1613 "cp1254",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1614 "cp1255",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1615 "cp1256",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1616 "cp1257",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1617 "cp1258",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1618 "cp424",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1619 "cp437",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1620 "cp500",
62572 1d1797af4622 Merged revisions 81499,81506 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 62367 diff changeset	1621 "cp720",
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1622 "cp737",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1623 "cp775",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1624 "cp850",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1625 "cp852",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1626 "cp855",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1627 "cp856",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1628 "cp857",
62572 1d1797af4622 Merged revisions 81499,81506 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 62367 diff changeset	1629 "cp858",
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1630 "cp860",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1631 "cp861",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1632 "cp862",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1633 "cp863",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1634 "cp864",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1635 "cp865",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1636 "cp866",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1637 "cp869",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1638 "cp874",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1639 "cp875",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1640 "cp932",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1641 "cp949",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1642 "cp950",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1643 "euc_jis_2004",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1644 "euc_jisx0213",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1645 "euc_jp",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1646 "euc_kr",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1647 "gb18030",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1648 "gb2312",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1649 "gbk",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1650 "hp_roman8",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1651 "hz",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1652 "idna",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1653 "iso2022_jp",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1654 "iso2022_jp_1",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1655 "iso2022_jp_2",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1656 "iso2022_jp_2004",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1657 "iso2022_jp_3",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1658 "iso2022_jp_ext",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1659 "iso2022_kr",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1660 "iso8859_1",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1661 "iso8859_10",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1662 "iso8859_11",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1663 "iso8859_13",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1664 "iso8859_14",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1665 "iso8859_15",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1666 "iso8859_16",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1667 "iso8859_2",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1668 "iso8859_3",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1669 "iso8859_4",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1670 "iso8859_5",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1671 "iso8859_6",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1672 "iso8859_7",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1673 "iso8859_8",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1674 "iso8859_9",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1675 "johab",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1676 "koi8_r",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1677 "koi8_u",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1678 "latin_1",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1679 "mac_cyrillic",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1680 "mac_greek",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1681 "mac_iceland",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1682 "mac_latin2",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1683 "mac_roman",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1684 "mac_turkish",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1685 "palmos",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1686 "ptcp154",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1687 "punycode",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1688 "raw_unicode_escape",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1689 "shift_jis",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1690 "shift_jis_2004",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1691 "shift_jisx0213",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1692 "tis_620",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1693 "unicode_escape",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1694 "unicode_internal",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1695 "utf_16",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1696 "utf_16_be",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1697 "utf_16_le",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1698 "utf_7",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1699 "utf_8",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1700 ]
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1701
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1702 if hasattr(codecs, "mbcs_encode"):
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1703 all_unicode_encodings.append("mbcs")
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1704
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1705 # The following encoding is not tested, because it's not supposed
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1706 # to work:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1707 # "undefined"
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1708
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1709 # The following encodings don't work in stateful mode
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1710 broken_unicode_with_streams = [
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1711 "punycode",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1712 "unicode_internal"
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1713 ]
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	1714 broken_incremental_coders = broken_unicode_with_streams + [
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	1715 "idna",
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	1716 ]
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1717
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	1718 class BasicUnicodeTest(unittest.TestCase, MixInCheckStateHandling):
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1719 def test_basics(self):
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1720 s = "abc123" # all codecs should be able to encode these
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1721 for encoding in all_unicode_encodings:
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1722 name = codecs.lookup(encoding).name
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1723 if encoding.endswith("_codec"):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1724 name += "_codec"
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1725 elif encoding == "latin_1":
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1726 name = "latin_1"
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1727 self.assertEqual(encoding.replace("_", "-"), name.replace("_", "-"))
73579 ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	1728
73600 fe2be7d35660 #13406: fix more deprecation warnings and move the deprecation of unicode-internal earlier in the code. Ezio Melotti <ezio.melotti@gmail.com> parents: 73589 diff changeset	1729 with support.check_warnings():
73579 ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	1730 # unicode-internal has been deprecated
ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	1731 (b, size) = codecs.getencoder(encoding)(s)
ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	1732 self.assertEqual(size, len(s), "%r != %r (encoding=%r)" % (size, len(s), encoding))
ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	1733 (chars, size) = codecs.getdecoder(encoding)(b)
ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	1734 self.assertEqual(chars, s, "%r != %r (encoding=%r)" % (chars, s, encoding))
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1735
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1736 if encoding not in broken_unicode_with_streams:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1737 # check stream reader/writer
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1738 q = Queue(b"")
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1739 writer = codecs.getwriter(encoding)(q)
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1740 encodedresult = b""
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1741 for c in s:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1742 writer.write(c)
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43577 diff changeset	1743 chunk = q.read()
55593 0bb232078b8c Merged revisions 73715 via svnmerge from Georg Brandl <georg@python.org> parents: 54988 diff changeset	1744 self.assertTrue(type(chunk) is bytes, type(chunk))
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43577 diff changeset	1745 encodedresult += chunk
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1746 q = Queue(b"")
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1747 reader = codecs.getreader(encoding)(q)
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1748 decodedresult = ""
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1749 for c in encodedresult:
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1750 q.write(bytes([c]))
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1751 decodedresult += reader.read()
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1752 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1753
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1754 if encoding not in broken_incremental_coders:
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1755 # check incremental decoder/encoder (fetched via the Python
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1756 # and C API) and iterencode()/iterdecode()
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1757 try:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1758 encoder = codecs.getincrementalencoder(encoding)()
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1759 cencoder = _testcapi.codec_incrementalencoder(encoding)
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1760 except LookupError: # no IncrementalEncoder
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1761 pass
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1762 else:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1763 # check incremental decoder/encoder
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1764 encodedresult = b""
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1765 for c in s:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1766 encodedresult += encoder.encode(c)
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1767 encodedresult += encoder.encode("", True)
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1768 decoder = codecs.getincrementaldecoder(encoding)()
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1769 decodedresult = ""
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1770 for c in encodedresult:
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1771 decodedresult += decoder.decode(bytes([c]))
41757 13a6c6bef609 Make test_codecs work. The CJK codecs now use bytes instead of str8 for Guido van Rossum <guido@python.org> parents: 41750 diff changeset	1772 decodedresult += decoder.decode(b"", True)
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1773 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1774
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1775 # check C API
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1776 encodedresult = b""
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1777 for c in s:
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1778 encodedresult += cencoder.encode(c)
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1779 encodedresult += cencoder.encode("", True)
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1780 cdecoder = _testcapi.codec_incrementaldecoder(encoding)
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1781 decodedresult = ""
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	1782 for c in encodedresult:
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1783 decodedresult += cdecoder.decode(bytes([c]))
41757 13a6c6bef609 Make test_codecs work. The CJK codecs now use bytes instead of str8 for Guido van Rossum <guido@python.org> parents: 41750 diff changeset	1784 decodedresult += cdecoder.decode(b"", True)
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1785 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1786
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1787 # check iterencode()/iterdecode()
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1788 result = "".join(codecs.iterdecode(codecs.iterencode(s, encoding), encoding))
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1789 self.assertEqual(result, s, "%r != %r (encoding=%r)" % (result, s, encoding))
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1790
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1791 # check iterencode()/iterdecode() with empty string
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1792 result = "".join(codecs.iterdecode(codecs.iterencode("", encoding), encoding))
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1793 self.assertEqual(result, "")
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1794
62367 283d9472df4e Issue #850997: mbcs encoding (Windows only) handles errors argument: strict Victor Stinner <victor.stinner@haypocalc.com> parents: 62256 diff changeset	1795 if encoding not in ("idna", "mbcs"):
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1796 # check incremental decoder/encoder with errors argument
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1797 try:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1798 encoder = codecs.getincrementalencoder(encoding)("ignore")
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1799 cencoder = _testcapi.codec_incrementalencoder(encoding, "ignore")
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1800 except LookupError: # no IncrementalEncoder
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1801 pass
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1802 else:
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1803 encodedresult = b"".join(encoder.encode(c) for c in s)
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1804 decoder = codecs.getincrementaldecoder(encoding)("ignore")
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1805 decodedresult = "".join(decoder.decode(bytes([c])) for c in encodedresult)
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1806 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1807
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1808 encodedresult = b"".join(cencoder.encode(c) for c in s)
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1809 cdecoder = _testcapi.codec_incrementaldecoder(encoding, "ignore")
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1810 decodedresult = "".join(cdecoder.decode(bytes([c])) for c in encodedresult)
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1811 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1812
34750 679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1813 def test_seek(self):
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1814 # all codecs should be able to encode these
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1815 s = "%s\n%s\n" % (100"abc123", 100"def456")
34750 679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1816 for encoding in all_unicode_encodings:
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1817 if encoding == "idna": # FIXME: See SF bug #1163178
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1818 continue
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1819 if encoding in broken_unicode_with_streams:
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1820 continue
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	1821 reader = codecs.getreader(encoding)(io.BytesIO(s.encode(encoding)))
41642 a6eb6acfe04a Merged revisions 55007-55179 via svnmerge from Guido van Rossum <guido@python.org> parents: 41606 diff changeset	1822 for t in range(5):
34750 679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1823 # Test that calling seek resets the internal codec state and buffers
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1824 reader.seek(0, 0)
41757 13a6c6bef609 Make test_codecs work. The CJK codecs now use bytes instead of str8 for Guido van Rossum <guido@python.org> parents: 41750 diff changeset	1825 data = reader.read()
13a6c6bef609 Make test_codecs work. The CJK codecs now use bytes instead of str8 for Guido van Rossum <guido@python.org> parents: 41750 diff changeset	1826 self.assertEqual(s, data)
34750 679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1827
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1828 def test_bad_decode_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1829 for encoding in all_unicode_encodings:
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1830 decoder = codecs.getdecoder(encoding)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1831 self.assertRaises(TypeError, decoder)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1832 if encoding not in ("idna", "punycode"):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1833 self.assertRaises(TypeError, decoder, 42)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1834
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1835 def test_bad_encode_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1836 for encoding in all_unicode_encodings:
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1837 encoder = codecs.getencoder(encoding)
73600 fe2be7d35660 #13406: fix more deprecation warnings and move the deprecation of unicode-internal earlier in the code. Ezio Melotti <ezio.melotti@gmail.com> parents: 73589 diff changeset	1838 with support.check_warnings():
fe2be7d35660 #13406: fix more deprecation warnings and move the deprecation of unicode-internal earlier in the code. Ezio Melotti <ezio.melotti@gmail.com> parents: 73589 diff changeset	1839 # unicode-internal has been deprecated
fe2be7d35660 #13406: fix more deprecation warnings and move the deprecation of unicode-internal earlier in the code. Ezio Melotti <ezio.melotti@gmail.com> parents: 73589 diff changeset	1840 self.assertRaises(TypeError, encoder)
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1841
39442 654c380cf8b9 Merged revisions 46753-51188 via svnmerge from Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	1842 def test_encoding_map_type_initialized(self):
654c380cf8b9 Merged revisions 46753-51188 via svnmerge from Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	1843 from encodings import cp1140
654c380cf8b9 Merged revisions 46753-51188 via svnmerge from Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	1844 # This used to crash, we are only verifying there's no crash.
654c380cf8b9 Merged revisions 46753-51188 via svnmerge from Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	1845 table_type = type(cp1140.encoding_table)
654c380cf8b9 Merged revisions 46753-51188 via svnmerge from Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	1846 self.assertEqual(table_type, table_type)
654c380cf8b9 Merged revisions 46753-51188 via svnmerge from Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	1847
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	1848 def test_decoder_state(self):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	1849 # Check that getstate() and setstate() handle the state properly
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1850 u = "abc123"
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	1851 for encoding in all_unicode_encodings:
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	1852 if encoding not in broken_incremental_coders:
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	1853 self.check_state_handling_decode(encoding, u, u.encode(encoding))
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	1854 self.check_state_handling_encode(encoding, u, u.encode(encoding))
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	1855
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1856 class CharmapTest(unittest.TestCase):
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1857 def test_decode_with_string_map(self):
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1858 self.assertEqual(
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1859 codecs.charmap_decode(b"\x00\x01\x02", "strict", "abc"),
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1860 ("abc", 3)
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1861 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1862
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1863 self.assertEqual(
79108 c64dec45d46f Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78207 79107 diff changeset	1864 codecs.charmap_decode(b"\x00\x01\x02", "strict", "\U0010FFFFbc"),
c64dec45d46f Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78207 79107 diff changeset	1865 ("\U0010FFFFbc", 3)
c64dec45d46f Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78207 79107 diff changeset	1866 )
c64dec45d46f Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78207 79107 diff changeset	1867
79107 620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1868 self.assertRaises(UnicodeDecodeError,
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1869 codecs.charmap_decode, b"\x00\x01\x02", "strict", "ab"
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1870 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1871
81518 13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1872 self.assertRaises(UnicodeDecodeError,
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1873 codecs.charmap_decode, b"\x00\x01\x02", "strict", "ab\ufffe"
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1874 )
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1875
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1876 self.assertEqual(
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1877 codecs.charmap_decode(b"\x00\x01\x02", "replace", "ab"),
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1878 ("ab\ufffd", 3)
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1879 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1880
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1881 self.assertEqual(
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1882 codecs.charmap_decode(b"\x00\x01\x02", "replace", "ab\ufffe"),
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1883 ("ab\ufffd", 3)
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1884 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1885
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1886 self.assertEqual(
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1887 codecs.charmap_decode(b"\x00\x01\x02", "ignore", "ab"),
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1888 ("ab", 3)
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1889 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1890
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1891 self.assertEqual(
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1892 codecs.charmap_decode(b"\x00\x01\x02", "ignore", "ab\ufffe"),
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1893 ("ab", 3)
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1894 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1895
41642 a6eb6acfe04a Merged revisions 55007-55179 via svnmerge from Guido van Rossum <guido@python.org> parents: 41606 diff changeset	1896 allbytes = bytes(range(256))
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1897 self.assertEqual(
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1898 codecs.charmap_decode(allbytes, "ignore", ""),
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1899 ("", len(allbytes))
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1900 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1901
79107 620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1902 def test_decode_with_int2str_map(self):
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1903 self.assertEqual(
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1904 codecs.charmap_decode(b"\x00\x01\x02", "strict",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1905 {0: 'a', 1: 'b', 2: 'c'}),
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1906 ("abc", 3)
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1907 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1908
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1909 self.assertEqual(
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1910 codecs.charmap_decode(b"\x00\x01\x02", "strict",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1911 {0: 'Aa', 1: 'Bb', 2: 'Cc'}),
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1912 ("AaBbCc", 3)
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1913 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1914
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1915 self.assertEqual(
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1916 codecs.charmap_decode(b"\x00\x01\x02", "strict",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1917 {0: '\U0010FFFF', 1: 'b', 2: 'c'}),
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1918 ("\U0010FFFFbc", 3)
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1919 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1920
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1921 self.assertEqual(
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1922 codecs.charmap_decode(b"\x00\x01\x02", "strict",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1923 {0: 'a', 1: 'b', 2: ''}),
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1924 ("ab", 3)
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1925 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1926
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1927 self.assertRaises(UnicodeDecodeError,
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1928 codecs.charmap_decode, b"\x00\x01\x02", "strict",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1929 {0: 'a', 1: 'b'}
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1930 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1931
81518 13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1932 self.assertRaises(UnicodeDecodeError,
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1933 codecs.charmap_decode, b"\x00\x01\x02", "strict",
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1934 {0: 'a', 1: 'b', 2: None}
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1935 )
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1936
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1937 # Issue #14850
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1938 self.assertRaises(UnicodeDecodeError,
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1939 codecs.charmap_decode, b"\x00\x01\x02", "strict",
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1940 {0: 'a', 1: 'b', 2: '\ufffe'}
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1941 )
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1942
79107 620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1943 self.assertEqual(
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1944 codecs.charmap_decode(b"\x00\x01\x02", "replace",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1945 {0: 'a', 1: 'b'}),
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1946 ("ab\ufffd", 3)
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1947 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1948
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1949 self.assertEqual(
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1950 codecs.charmap_decode(b"\x00\x01\x02", "replace",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1951 {0: 'a', 1: 'b', 2: None}),
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1952 ("ab\ufffd", 3)
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1953 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1954
81518 13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1955 # Issue #14850
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1956 self.assertEqual(
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1957 codecs.charmap_decode(b"\x00\x01\x02", "replace",
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1958 {0: 'a', 1: 'b', 2: '\ufffe'}),
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1959 ("ab\ufffd", 3)
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1960 )
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1961
79107 620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1962 self.assertEqual(
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1963 codecs.charmap_decode(b"\x00\x01\x02", "ignore",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1964 {0: 'a', 1: 'b'}),
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1965 ("ab", 3)
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1966 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1967
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1968 self.assertEqual(
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1969 codecs.charmap_decode(b"\x00\x01\x02", "ignore",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1970 {0: 'a', 1: 'b', 2: None}),
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1971 ("ab", 3)
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1972 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1973
81518 13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1974 # Issue #14850
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1975 self.assertEqual(
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1976 codecs.charmap_decode(b"\x00\x01\x02", "ignore",
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1977 {0: 'a', 1: 'b', 2: '\ufffe'}),
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1978 ("ab", 3)
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1979 )
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	1980
79107 620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1981 allbytes = bytes(range(256))
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1982 self.assertEqual(
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1983 codecs.charmap_decode(allbytes, "ignore", {}),
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1984 ("", len(allbytes))
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1985 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1986
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1987 def test_decode_with_int2int_map(self):
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1988 a = ord('a')
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1989 b = ord('b')
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1990 c = ord('c')
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1991
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1992 self.assertEqual(
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1993 codecs.charmap_decode(b"\x00\x01\x02", "strict",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1994 {0: a, 1: b, 2: c}),
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1995 ("abc", 3)
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1996 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1997
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1998 # Issue #15379
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1999 self.assertEqual(
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	2000 codecs.charmap_decode(b"\x00\x01\x02", "strict",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	2001 {0: 0x10FFFF, 1: b, 2: c}),
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	2002 ("\U0010FFFFbc", 3)
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	2003 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	2004
79108 c64dec45d46f Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78207 79107 diff changeset	2005 self.assertEqual(
c64dec45d46f Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78207 79107 diff changeset	2006 codecs.charmap_decode(b"\x00\x01\x02", "strict",
c64dec45d46f Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78207 79107 diff changeset	2007 {0: sys.maxunicode, 1: b, 2: c}),
c64dec45d46f Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78207 79107 diff changeset	2008 (chr(sys.maxunicode) + "bc", 3)
c64dec45d46f Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78207 79107 diff changeset	2009 )
c64dec45d46f Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78207 79107 diff changeset	2010
79107 620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	2011 self.assertRaises(TypeError,
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	2012 codecs.charmap_decode, b"\x00\x01\x02", "strict",
79108 c64dec45d46f Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78207 79107 diff changeset	2013 {0: sys.maxunicode + 1, 1: b, 2: c}
79107 620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	2014 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	2015
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	2016 self.assertRaises(UnicodeDecodeError,
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	2017 codecs.charmap_decode, b"\x00\x01\x02", "strict",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	2018 {0: a, 1: b},
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	2019 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	2020
81518 13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	2021 self.assertRaises(UnicodeDecodeError,
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	2022 codecs.charmap_decode, b"\x00\x01\x02", "strict",
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	2023 {0: a, 1: b, 2: 0xFFFE},
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	2024 )
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	2025
79107 620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	2026 self.assertEqual(
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	2027 codecs.charmap_decode(b"\x00\x01\x02", "replace",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	2028 {0: a, 1: b}),
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	2029 ("ab\ufffd", 3)
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	2030 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	2031
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	2032 self.assertEqual(
81518 13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	2033 codecs.charmap_decode(b"\x00\x01\x02", "replace",
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	2034 {0: a, 1: b, 2: 0xFFFE}),
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	2035 ("ab\ufffd", 3)
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	2036 )
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	2037
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	2038 self.assertEqual(
79107 620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	2039 codecs.charmap_decode(b"\x00\x01\x02", "ignore",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	2040 {0: a, 1: b}),
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	2041 ("ab", 3)
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	2042 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	2043
81518 13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	2044 self.assertEqual(
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	2045 codecs.charmap_decode(b"\x00\x01\x02", "ignore",
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	2046 {0: a, 1: b, 2: 0xFFFE}),
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	2047 ("ab", 3)
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	2048 )
13cd78a2a17b Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" Serhiy Storchaka <storchaka@gmail.com> parents: 81381 diff changeset	2049
79107 620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	2050
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	2051 class WithStmtTest(unittest.TestCase):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	2052 def test_encodedfile(self):
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	2053 f = io.BytesIO(b"\xc3\xbc")
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	2054 with codecs.EncodedFile(f, "latin-1", "utf-8") as ef:
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	2055 self.assertEqual(ef.read(), b"\xfc")
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	2056
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	2057 def test_streamreaderwriter(self):
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	2058 f = io.BytesIO(b"\xc3\xbc")
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	2059 info = codecs.lookup("utf-8")
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	2060 with codecs.StreamReaderWriter(f, info.streamreader,
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	2061 info.streamwriter, 'strict') as srw:
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	2062 self.assertEqual(srw.read(), "\xfc")
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	2063
50975 ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	2064 class TypesTest(unittest.TestCase):
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	2065 def test_decode_unicode(self):
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	2066 # Most decoders don't accept unicode input
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	2067 decoders = [
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	2068 codecs.utf_7_decode,
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	2069 codecs.utf_8_decode,
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	2070 codecs.utf_16_le_decode,
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	2071 codecs.utf_16_be_decode,
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	2072 codecs.utf_16_ex_decode,
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	2073 codecs.utf_32_decode,
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	2074 codecs.utf_32_le_decode,
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	2075 codecs.utf_32_be_decode,
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	2076 codecs.utf_32_ex_decode,
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	2077 codecs.latin_1_decode,
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	2078 codecs.ascii_decode,
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	2079 codecs.charmap_decode,
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	2080 ]
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	2081 if hasattr(codecs, "mbcs_decode"):
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	2082 decoders.append(codecs.mbcs_decode)
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	2083 for decoder in decoders:
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	2084 self.assertRaises(TypeError, decoder, "xxx")
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	2085
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	2086 def test_unicode_escape(self):
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	2087 # Escape-decoding an unicode string is supported ang gives the same
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	2088 # result as decoding the equivalent ASCII bytes string.
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	2089 self.assertEqual(codecs.unicode_escape_decode(r"\u1234"), ("\u1234", 6))
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	2090 self.assertEqual(codecs.unicode_escape_decode(br"\u1234"), ("\u1234", 6))
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	2091 self.assertEqual(codecs.raw_unicode_escape_decode(r"\u1234"), ("\u1234", 6))
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	2092 self.assertEqual(codecs.raw_unicode_escape_decode(br"\u1234"), ("\u1234", 6))
50975 ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	2093
73906 bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	2094 self.assertRaises(UnicodeDecodeError, codecs.unicode_escape_decode, br"\U00110000")
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	2095 self.assertEqual(codecs.unicode_escape_decode(r"\U00110000", "replace"), ("\ufffd", 10))
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	2096
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	2097 self.assertRaises(UnicodeDecodeError, codecs.raw_unicode_escape_decode, br"\U00110000")
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	2098 self.assertEqual(codecs.raw_unicode_escape_decode(r"\U00110000", "replace"), ("\ufffd", 10))
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	2099
81804 084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2100
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2101 class UnicodeEscapeTest(unittest.TestCase):
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2102 def test_empty(self):
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2103 self.assertEqual(codecs.unicode_escape_encode(""), (b"", 0))
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2104 self.assertEqual(codecs.unicode_escape_decode(b""), ("", 0))
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2105
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2106 def test_raw_encode(self):
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2107 encode = codecs.unicode_escape_encode
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2108 for b in range(32, 127):
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2109 if b != b'\\'[0]:
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2110 self.assertEqual(encode(chr(b)), (bytes([b]), 1))
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2111
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2112 def test_raw_decode(self):
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2113 decode = codecs.unicode_escape_decode
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2114 for b in range(256):
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2115 if b != b'\\'[0]:
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2116 self.assertEqual(decode(bytes([b]) + b'0'), (chr(b) + '0', 2))
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2117
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2118 def test_escape_encode(self):
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2119 encode = codecs.unicode_escape_encode
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2120 check = coding_checker(self, encode)
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2121 check('\t', br'\t')
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2122 check('\n', br'\n')
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2123 check('\r', br'\r')
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2124 check('\\', br'\\')
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2125 for b in range(32):
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2126 if chr(b) not in '\t\n\r':
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2127 check(chr(b), ('\\x%02x' % b).encode())
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2128 for b in range(127, 256):
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2129 check(chr(b), ('\\x%02x' % b).encode())
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2130 check('\u20ac', br'\u20ac')
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2131 check('\U0001d120', br'\U0001d120')
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2132
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2133 def test_escape_decode(self):
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2134 decode = codecs.unicode_escape_decode
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2135 check = coding_checker(self, decode)
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2136 check(b"[\\\n]", "[]")
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2137 check(br'[\"]', '["]')
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2138 check(br"[\']", "[']")
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2139 check(br"[\\]", r"[\]")
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2140 check(br"[\a]", "[\x07]")
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2141 check(br"[\b]", "[\x08]")
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2142 check(br"[\t]", "[\x09]")
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2143 check(br"[\n]", "[\x0a]")
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2144 check(br"[\v]", "[\x0b]")
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2145 check(br"[\f]", "[\x0c]")
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2146 check(br"[\r]", "[\x0d]")
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2147 check(br"[\7]", "[\x07]")
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2148 check(br"[\8]", r"[\8]")
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2149 check(br"[\78]", "[\x078]")
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2150 check(br"[\41]", "[!]")
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2151 check(br"[\418]", "[!8]")
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2152 check(br"[\101]", "[A]")
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2153 check(br"[\1010]", "[A0]")
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2154 check(br"[\x41]", "[A]")
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2155 check(br"[\x410]", "[A0]")
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2156 check(br"\u20ac", "\u20ac")
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2157 check(br"\U0001d120", "\U0001d120")
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2158 for b in range(256):
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2159 if b not in b'\n"\'\\abtnvfr01234567xuUN':
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2160 check(b'\\' + bytes([b]), '\\' + chr(b))
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2161
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2162 def test_decode_errors(self):
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2163 decode = codecs.unicode_escape_decode
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2164 for c, d in (b'x', 2), (b'u', 4), (b'U', 4):
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2165 for i in range(d):
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2166 self.assertRaises(UnicodeDecodeError, decode,
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2167 b"\\" + c + b"0"*i)
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2168 self.assertRaises(UnicodeDecodeError, decode,
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2169 b"[\\" + c + b"0"*i + b"]")
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2170 data = b"[\\" + c + b"0"i + b"]\\" + c + b"0"i
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2171 self.assertEqual(decode(data, "ignore"), ("[]", len(data)))
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2172 self.assertEqual(decode(data, "replace"),
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2173 ("[\ufffd]\ufffd", len(data)))
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2174 self.assertRaises(UnicodeDecodeError, decode, br"\U00110000")
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2175 self.assertEqual(decode(br"\U00110000", "ignore"), ("", 10))
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2176 self.assertEqual(decode(br"\U00110000", "replace"), ("\ufffd", 10))
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2177
084bec5443d6 Issue #16979: Fix error handling bugs in the unicode-escape-decode decoder. Serhiy Storchaka <storchaka@gmail.com> parents: 81738 diff changeset	2178
81818 eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2179 class RawUnicodeEscapeTest(unittest.TestCase):
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2180 def test_empty(self):
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2181 self.assertEqual(codecs.raw_unicode_escape_encode(""), (b"", 0))
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2182 self.assertEqual(codecs.raw_unicode_escape_decode(b""), ("", 0))
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2183
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2184 def test_raw_encode(self):
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2185 encode = codecs.raw_unicode_escape_encode
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2186 for b in range(256):
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2187 self.assertEqual(encode(chr(b)), (bytes([b]), 1))
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2188
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2189 def test_raw_decode(self):
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2190 decode = codecs.raw_unicode_escape_decode
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2191 for b in range(256):
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2192 self.assertEqual(decode(bytes([b]) + b'0'), (chr(b) + '0', 2))
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2193
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2194 def test_escape_encode(self):
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2195 encode = codecs.raw_unicode_escape_encode
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2196 check = coding_checker(self, encode)
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2197 for b in range(256):
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2198 if b not in b'uU':
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2199 check('\\' + chr(b), b'\\' + bytes([b]))
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2200 check('\u20ac', br'\u20ac')
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2201 check('\U0001d120', br'\U0001d120')
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2202
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2203 def test_escape_decode(self):
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2204 decode = codecs.raw_unicode_escape_decode
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2205 check = coding_checker(self, decode)
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2206 for b in range(256):
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2207 if b not in b'uU':
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2208 check(b'\\' + bytes([b]), '\\' + chr(b))
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2209 check(br"\u20ac", "\u20ac")
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2210 check(br"\U0001d120", "\U0001d120")
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2211
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2212 def test_decode_errors(self):
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2213 decode = codecs.raw_unicode_escape_decode
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2214 for c, d in (b'u', 4), (b'U', 4):
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2215 for i in range(d):
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2216 self.assertRaises(UnicodeDecodeError, decode,
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2217 b"\\" + c + b"0"*i)
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2218 self.assertRaises(UnicodeDecodeError, decode,
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2219 b"[\\" + c + b"0"*i + b"]")
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2220 data = b"[\\" + c + b"0"i + b"]\\" + c + b"0"i
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2221 self.assertEqual(decode(data, "ignore"), ("[]", len(data)))
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2222 self.assertEqual(decode(data, "replace"),
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2223 ("[\ufffd]\ufffd", len(data)))
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2224 self.assertRaises(UnicodeDecodeError, decode, br"\U00110000")
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2225 self.assertEqual(decode(br"\U00110000", "ignore"), ("", 10))
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2226 self.assertEqual(decode(br"\U00110000", "replace"), ("\ufffd", 10))
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2227
eb4ef5c2875d Add tests for raw-unicode-escape codec. Serhiy Storchaka <storchaka@gmail.com> parents: 81814 diff changeset	2228
53966 7c29bfbfd352 Rename utf8b error handler to surrogateescape. Martin v. Löwis <martin@v.loewis.de> parents: 53965 diff changeset	2229 class SurrogateEscapeTest(unittest.TestCase):
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	2230
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	2231 def test_utf8(self):
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	2232 # Bad byte
53966 7c29bfbfd352 Rename utf8b error handler to surrogateescape. Martin v. Löwis <martin@v.loewis.de> parents: 53965 diff changeset	2233 self.assertEqual(b"foo\x80bar".decode("utf-8", "surrogateescape"),
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	2234 "foo\udc80bar")
53966 7c29bfbfd352 Rename utf8b error handler to surrogateescape. Martin v. Löwis <martin@v.loewis.de> parents: 53965 diff changeset	2235 self.assertEqual("foo\udc80bar".encode("utf-8", "surrogateescape"),
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	2236 b"foo\x80bar")
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	2237 # bad-utf-8 encoded surrogate
53966 7c29bfbfd352 Rename utf8b error handler to surrogateescape. Martin v. Löwis <martin@v.loewis.de> parents: 53965 diff changeset	2238 self.assertEqual(b"\xed\xb0\x80".decode("utf-8", "surrogateescape"),
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	2239 "\udced\udcb0\udc80")
53966 7c29bfbfd352 Rename utf8b error handler to surrogateescape. Martin v. Löwis <martin@v.loewis.de> parents: 53965 diff changeset	2240 self.assertEqual("\udced\udcb0\udc80".encode("utf-8", "surrogateescape"),
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	2241 b"\xed\xb0\x80")
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	2242
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	2243 def test_ascii(self):
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	2244 # bad byte
53966 7c29bfbfd352 Rename utf8b error handler to surrogateescape. Martin v. Löwis <martin@v.loewis.de> parents: 53965 diff changeset	2245 self.assertEqual(b"foo\x80bar".decode("ascii", "surrogateescape"),
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	2246 "foo\udc80bar")
53966 7c29bfbfd352 Rename utf8b error handler to surrogateescape. Martin v. Löwis <martin@v.loewis.de> parents: 53965 diff changeset	2247 self.assertEqual("foo\udc80bar".encode("ascii", "surrogateescape"),
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	2248 b"foo\x80bar")
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	2249
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	2250 def test_charmap(self):
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	2251 # bad byte: \xa5 is unmapped in iso-8859-3
53966 7c29bfbfd352 Rename utf8b error handler to surrogateescape. Martin v. Löwis <martin@v.loewis.de> parents: 53965 diff changeset	2252 self.assertEqual(b"foo\xa5bar".decode("iso-8859-3", "surrogateescape"),
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	2253 "foo\udca5bar")
53966 7c29bfbfd352 Rename utf8b error handler to surrogateescape. Martin v. Löwis <martin@v.loewis.de> parents: 53965 diff changeset	2254 self.assertEqual("foo\udca5bar".encode("iso-8859-3", "surrogateescape"),
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	2255 b"foo\xa5bar")
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	2256
54988 29fac8848489 Merged revisions 73698 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53966 diff changeset	2257 def test_latin1(self):
29fac8848489 Merged revisions 73698 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53966 diff changeset	2258 # Issue6373
68142 be6c38d1817b Normalize the encoding names for Latin-1 and UTF-8 to Marc-André Lemburg <mal@egenix.com> parents: 66837 diff changeset	2259 self.assertEqual("\udce4\udceb\udcef\udcf6\udcfc".encode("latin-1", "surrogateescape"),
54988 29fac8848489 Merged revisions 73698 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53966 diff changeset	2260 b"\xe4\xeb\xef\xf6\xfc")
29fac8848489 Merged revisions 73698 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53966 diff changeset	2261
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	2262
61850 846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	2263 class BomTest(unittest.TestCase):
846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	2264 def test_seek0(self):
846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	2265 data = "1234567890"
846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	2266 tests = ("utf-16",
846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	2267 "utf-16-le",
846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	2268 "utf-16-be",
846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	2269 "utf-32",
846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	2270 "utf-32-le",
846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	2271 "utf-32-be")
70292 b538445a9df8 test_codecs now removes the temporay file (created by the test) Victor Stinner <victor.stinner@haypocalc.com> parents: 66340 diff changeset	2272 self.addCleanup(support.unlink, support.TESTFN)
61850 846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	2273 for encoding in tests:
61863 cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2274 # Check if the BOM is written only once
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2275 with codecs.open(support.TESTFN, 'w+', encoding=encoding) as f:
61850 846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	2276 f.write(data)
846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	2277 f.write(data)
846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	2278 f.seek(0)
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	2279 self.assertEqual(f.read(), data * 2)
61850 846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	2280 f.seek(0)
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	2281 self.assertEqual(f.read(), data * 2)
61850 846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	2282
61863 cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2283 # Check that the BOM is written after a seek(0)
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2284 with codecs.open(support.TESTFN, 'w+', encoding=encoding) as f:
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2285 f.write(data[0])
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	2286 self.assertNotEqual(f.tell(), 0)
61863 cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2287 f.seek(0)
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2288 f.write(data)
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2289 f.seek(0)
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	2290 self.assertEqual(f.read(), data)
61863 cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2291
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2292 # (StreamWriter) Check that the BOM is written after a seek(0)
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2293 with codecs.open(support.TESTFN, 'w+', encoding=encoding) as f:
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2294 f.writer.write(data[0])
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	2295 self.assertNotEqual(f.writer.tell(), 0)
61863 cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2296 f.writer.seek(0)
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2297 f.writer.write(data)
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2298 f.seek(0)
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	2299 self.assertEqual(f.read(), data)
61863 cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2300
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2301 # Check that the BOM is not written after a seek() at a position
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2302 # different than the start
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2303 with codecs.open(support.TESTFN, 'w+', encoding=encoding) as f:
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2304 f.write(data)
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2305 f.seek(f.tell())
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2306 f.write(data)
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2307 f.seek(0)
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	2308 self.assertEqual(f.read(), data * 2)
61863 cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2309
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2310 # (StreamWriter) Check that the BOM is not written after a seek()
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2311 # at a position different than the start
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2312 with codecs.open(support.TESTFN, 'w+', encoding=encoding) as f:
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2313 f.writer.write(data)
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2314 f.writer.seek(f.writer.tell())
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2315 f.writer.write(data)
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2316 f.seek(0)
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	2317 self.assertEqual(f.read(), data * 2)
61863 cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	2318
61850 846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	2319
66605 7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2320 bytes_transform_encodings = [
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2321 "base64_codec",
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2322 "uu_codec",
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2323 "quopri_codec",
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2324 "hex_codec",
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2325 ]
87388 5e960d2c2156 Close #7475: Restore binary & text transform codecs Nick Coghlan <ncoghlan@gmail.com> parents: 87338 diff changeset	2326
5e960d2c2156 Close #7475: Restore binary & text transform codecs Nick Coghlan <ncoghlan@gmail.com> parents: 87338 diff changeset	2327 transform_aliases = {
5e960d2c2156 Close #7475: Restore binary & text transform codecs Nick Coghlan <ncoghlan@gmail.com> parents: 87338 diff changeset	2328 "base64_codec": ["base64", "base_64"],
5e960d2c2156 Close #7475: Restore binary & text transform codecs Nick Coghlan <ncoghlan@gmail.com> parents: 87338 diff changeset	2329 "uu_codec": ["uu"],
5e960d2c2156 Close #7475: Restore binary & text transform codecs Nick Coghlan <ncoghlan@gmail.com> parents: 87338 diff changeset	2330 "quopri_codec": ["quopri", "quoted_printable", "quotedprintable"],
5e960d2c2156 Close #7475: Restore binary & text transform codecs Nick Coghlan <ncoghlan@gmail.com> parents: 87338 diff changeset	2331 "hex_codec": ["hex"],
5e960d2c2156 Close #7475: Restore binary & text transform codecs Nick Coghlan <ncoghlan@gmail.com> parents: 87338 diff changeset	2332 "rot_13": ["rot13"],
5e960d2c2156 Close #7475: Restore binary & text transform codecs Nick Coghlan <ncoghlan@gmail.com> parents: 87338 diff changeset	2333 }
5e960d2c2156 Close #7475: Restore binary & text transform codecs Nick Coghlan <ncoghlan@gmail.com> parents: 87338 diff changeset	2334
66605 7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2335 try:
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2336 import zlib
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2337 except ImportError:
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2338 pass
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2339 else:
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2340 bytes_transform_encodings.append("zlib_codec")
87388 5e960d2c2156 Close #7475: Restore binary & text transform codecs Nick Coghlan <ncoghlan@gmail.com> parents: 87338 diff changeset	2341 transform_aliases["zlib_codec"] = ["zip", "zlib"]
66605 7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2342 try:
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2343 import bz2
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2344 except ImportError:
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2345 pass
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2346 else:
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2347 bytes_transform_encodings.append("bz2_codec")
87388 5e960d2c2156 Close #7475: Restore binary & text transform codecs Nick Coghlan <ncoghlan@gmail.com> parents: 87338 diff changeset	2348 transform_aliases["bz2_codec"] = ["bz2"]
66605 7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2349
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2350 class TransformCodecTest(unittest.TestCase):
66837 a811f7c10e82 remove (un)transform methods Benjamin Peterson <benjamin@python.org> parents: 66797 diff changeset	2351
66605 7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2352 def test_basics(self):
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2353 binput = bytes(range(256))
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2354 for encoding in bytes_transform_encodings:
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2355 with self.subTest(encoding=encoding):
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2356 # generic codecs interface
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2357 (o, size) = codecs.getencoder(encoding)(binput)
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2358 self.assertEqual(size, len(binput))
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2359 (i, size) = codecs.getdecoder(encoding)(o)
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2360 self.assertEqual(size, len(o))
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2361 self.assertEqual(i, binput)
66605 7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2362
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2363 def test_read(self):
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2364 for encoding in bytes_transform_encodings:
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2365 with self.subTest(encoding=encoding):
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2366 sin = codecs.encode(b"\x80", encoding)
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2367 reader = codecs.getreader(encoding)(io.BytesIO(sin))
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2368 sout = reader.read()
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2369 self.assertEqual(sout, b"\x80")
66605 7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2370
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2371 def test_readline(self):
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2372 for encoding in bytes_transform_encodings:
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2373 if encoding in ['uu_codec', 'zlib_codec']:
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2374 continue
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2375 with self.subTest(encoding=encoding):
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2376 sin = codecs.encode(b"\x80", encoding)
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2377 reader = codecs.getreader(encoding)(io.BytesIO(sin))
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2378 sout = reader.readline()
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2379 self.assertEqual(sout, b"\x80")
66605 7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2380
85930 d90f25e1a705 Close #17839: support bytes-like objects in base64 module Nick Coghlan <ncoghlan@gmail.com> parents: 81820 diff changeset	2381 def test_buffer_api_usage(self):
d90f25e1a705 Close #17839: support bytes-like objects in base64 module Nick Coghlan <ncoghlan@gmail.com> parents: 81820 diff changeset	2382 # We check all the transform codecs accept memoryview input
d90f25e1a705 Close #17839: support bytes-like objects in base64 module Nick Coghlan <ncoghlan@gmail.com> parents: 81820 diff changeset	2383 # for encoding and decoding
d90f25e1a705 Close #17839: support bytes-like objects in base64 module Nick Coghlan <ncoghlan@gmail.com> parents: 81820 diff changeset	2384 # and also that they roundtrip correctly
d90f25e1a705 Close #17839: support bytes-like objects in base64 module Nick Coghlan <ncoghlan@gmail.com> parents: 81820 diff changeset	2385 original = b"12345\x80"
d90f25e1a705 Close #17839: support bytes-like objects in base64 module Nick Coghlan <ncoghlan@gmail.com> parents: 81820 diff changeset	2386 for encoding in bytes_transform_encodings:
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2387 with self.subTest(encoding=encoding):
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2388 data = original
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2389 view = memoryview(data)
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2390 data = codecs.encode(data, encoding)
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2391 view_encoded = codecs.encode(view, encoding)
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2392 self.assertEqual(view_encoded, data)
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2393 view = memoryview(data)
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2394 data = codecs.decode(data, encoding)
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2395 self.assertEqual(data, original)
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2396 view_decoded = codecs.decode(view, encoding)
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2397 self.assertEqual(view_decoded, data)
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2398
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2399 def test_text_to_binary_blacklists_binary_transforms(self):
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2400 # Check binary -> binary codecs give a good error for str input
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2401 bad_input = "bad input type"
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2402 for encoding in bytes_transform_encodings:
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2403 with self.subTest(encoding=encoding):
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2404 fmt = ( "{!r} is not a text encoding; "
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2405 "use codecs.encode to handle arbitrary codecs")
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2406 msg = fmt.format(encoding)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2407 with self.assertRaisesRegex(LookupError, msg) as failure:
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2408 bad_input.encode(encoding)
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2409 self.assertIsNone(failure.exception.__cause__)
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2410
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2411 def test_text_to_binary_blacklists_text_transforms(self):
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2412 # Check str.encode gives a good error message for str -> str codecs
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2413 msg = (r"^'rot_13' is not a text encoding; "
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2414 "use codecs.encode to handle arbitrary codecs")
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2415 with self.assertRaisesRegex(LookupError, msg):
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2416 "just an example message".encode("rot_13")
87267 04e1f701aeaa Also chain codec exceptions that allow weakrefs Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	2417
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2418 def test_binary_to_text_blacklists_binary_transforms(self):
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2419 # Check bytes.decode and bytearray.decode give a good error
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2420 # message for binary -> binary codecs
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2421 data = b"encode first to ensure we meet any format restrictions"
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2422 for encoding in bytes_transform_encodings:
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2423 with self.subTest(encoding=encoding):
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2424 encoded_data = codecs.encode(data, encoding)
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2425 fmt = (r"{!r} is not a text encoding; "
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2426 "use codecs.decode to handle arbitrary codecs")
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2427 msg = fmt.format(encoding)
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2428 with self.assertRaisesRegex(LookupError, msg):
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2429 encoded_data.decode(encoding)
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2430 with self.assertRaisesRegex(LookupError, msg):
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2431 bytearray(encoded_data).decode(encoding)
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2432
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2433 def test_binary_to_text_blacklists_text_transforms(self):
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2434 # Check str -> str codec gives a good error for binary input
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2435 for bad_input in (b"immutable", bytearray(b"mutable")):
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2436 with self.subTest(bad_input=bad_input):
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2437 msg = (r"^'rot_13' is not a text encoding; "
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2438 "use codecs.decode to handle arbitrary codecs")
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2439 with self.assertRaisesRegex(LookupError, msg) as failure:
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2440 bad_input.decode("rot_13")
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2441 self.assertIsNone(failure.exception.__cause__)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2442
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2443 def test_custom_zlib_error_is_wrapped(self):
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2444 # Check zlib codec gives a good error for malformed input
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2445 msg = "^decoding with 'zlib_codec' codec failed"
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2446 with self.assertRaisesRegex(Exception, msg) as failure:
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2447 codecs.decode(b"hello", "zlib_codec")
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2448 self.assertIsInstance(failure.exception.__cause__,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2449 type(failure.exception))
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2450
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2451 def test_custom_hex_error_is_wrapped(self):
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2452 # Check hex codec gives a good error for malformed input
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2453 msg = "^decoding with 'hex_codec' codec failed"
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2454 with self.assertRaisesRegex(Exception, msg) as failure:
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2455 codecs.decode(b"hello", "hex_codec")
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2456 self.assertIsInstance(failure.exception.__cause__,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2457 type(failure.exception))
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2458
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2459 # Unfortunately, the bz2 module throws OSError, which the codec
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2460 # machinery currently can't wrap :(
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2461
87388 5e960d2c2156 Close #7475: Restore binary & text transform codecs Nick Coghlan <ncoghlan@gmail.com> parents: 87338 diff changeset	2462 # Ensure codec aliases from http://bugs.python.org/issue7475 work
5e960d2c2156 Close #7475: Restore binary & text transform codecs Nick Coghlan <ncoghlan@gmail.com> parents: 87338 diff changeset	2463 def test_aliases(self):
5e960d2c2156 Close #7475: Restore binary & text transform codecs Nick Coghlan <ncoghlan@gmail.com> parents: 87338 diff changeset	2464 for codec_name, aliases in transform_aliases.items():
5e960d2c2156 Close #7475: Restore binary & text transform codecs Nick Coghlan <ncoghlan@gmail.com> parents: 87338 diff changeset	2465 expected_name = codecs.lookup(codec_name).name
5e960d2c2156 Close #7475: Restore binary & text transform codecs Nick Coghlan <ncoghlan@gmail.com> parents: 87338 diff changeset	2466 for alias in aliases:
5e960d2c2156 Close #7475: Restore binary & text transform codecs Nick Coghlan <ncoghlan@gmail.com> parents: 87338 diff changeset	2467 with self.subTest(alias=alias):
5e960d2c2156 Close #7475: Restore binary & text transform codecs Nick Coghlan <ncoghlan@gmail.com> parents: 87338 diff changeset	2468 info = codecs.lookup(alias)
5e960d2c2156 Close #7475: Restore binary & text transform codecs Nick Coghlan <ncoghlan@gmail.com> parents: 87338 diff changeset	2469 self.assertEqual(info.name, expected_name)
5e960d2c2156 Close #7475: Restore binary & text transform codecs Nick Coghlan <ncoghlan@gmail.com> parents: 87338 diff changeset	2470
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2471
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2472 # The codec system tries to wrap exceptions in order to ensure the error
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2473 # mentions the operation being performed and the codec involved. We
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2474 # currently only want this to happen for relatively stateless
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2475 # exceptions, where the only significant information they contain is their
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2476 # type and a single str argument.
87111 8c4d42c0dc8e Avoid triggering the refleak detector Nick Coghlan <ncoghlan@gmail.com> parents: 87109 diff changeset	2477
8c4d42c0dc8e Avoid triggering the refleak detector Nick Coghlan <ncoghlan@gmail.com> parents: 87109 diff changeset	2478 # Use a local codec registry to avoid appearing to leak objects when
8c4d42c0dc8e Avoid triggering the refleak detector Nick Coghlan <ncoghlan@gmail.com> parents: 87109 diff changeset	2479 # registering multiple seach functions
8c4d42c0dc8e Avoid triggering the refleak detector Nick Coghlan <ncoghlan@gmail.com> parents: 87109 diff changeset	2480 _TEST_CODECS = {}
8c4d42c0dc8e Avoid triggering the refleak detector Nick Coghlan <ncoghlan@gmail.com> parents: 87109 diff changeset	2481
8c4d42c0dc8e Avoid triggering the refleak detector Nick Coghlan <ncoghlan@gmail.com> parents: 87109 diff changeset	2482 def _get_test_codec(codec_name):
8c4d42c0dc8e Avoid triggering the refleak detector Nick Coghlan <ncoghlan@gmail.com> parents: 87109 diff changeset	2483 return _TEST_CODECS.get(codec_name)
8c4d42c0dc8e Avoid triggering the refleak detector Nick Coghlan <ncoghlan@gmail.com> parents: 87109 diff changeset	2484 codecs.register(_get_test_codec) # Returns None, not usable as a decorator
8c4d42c0dc8e Avoid triggering the refleak detector Nick Coghlan <ncoghlan@gmail.com> parents: 87109 diff changeset	2485
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2486 class ExceptionChainingTest(unittest.TestCase):
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2487
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2488 def setUp(self):
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2489 # There's no way to unregister a codec search function, so we just
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2490 # ensure we render this one fairly harmless after the test
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2491 # case finishes by using the test case repr as the codec name
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2492 # The codecs module normalizes codec names, although this doesn't
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2493 # appear to be formally documented...
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2494 # We also make sure we use a truly unique id for the custom codec
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2495 # to avoid issues with the codec cache when running these tests
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2496 # multiple times (e.g. when hunting for refleaks)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2497 unique_id = repr(self) + str(id(self))
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2498 self.codec_name = encodings.normalize_encoding(unique_id).lower()
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2499
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2500 # We store the object to raise on the instance because of a bad
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2501 # interaction between the codec caching (which means we can't
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2502 # recreate the codec entry) and regrtest refleak hunting (which
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2503 # runs the same test instance multiple times). This means we
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2504 # need to ensure the codecs call back in to the instance to find
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2505 # out which exception to raise rather than binding them in a
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2506 # closure to an object that may change on the next run
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2507 self.obj_to_raise = RuntimeError
85930 d90f25e1a705 Close #17839: support bytes-like objects in base64 module Nick Coghlan <ncoghlan@gmail.com> parents: 81820 diff changeset	2508
87111 8c4d42c0dc8e Avoid triggering the refleak detector Nick Coghlan <ncoghlan@gmail.com> parents: 87109 diff changeset	2509 def tearDown(self):
8c4d42c0dc8e Avoid triggering the refleak detector Nick Coghlan <ncoghlan@gmail.com> parents: 87109 diff changeset	2510 _TEST_CODECS.pop(self.codec_name, None)
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2511
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2512 def set_codec(self, encode, decode):
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2513 codec_info = codecs.CodecInfo(encode, decode,
87111 8c4d42c0dc8e Avoid triggering the refleak detector Nick Coghlan <ncoghlan@gmail.com> parents: 87109 diff changeset	2514 name=self.codec_name)
8c4d42c0dc8e Avoid triggering the refleak detector Nick Coghlan <ncoghlan@gmail.com> parents: 87109 diff changeset	2515 _TEST_CODECS[self.codec_name] = codec_info
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2516
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2517 @contextlib.contextmanager
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2518 def assertWrapped(self, operation, exc_type, msg):
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2519 full_msg = r"{} with {!r} codec failed ${}: {}$".format(
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2520 operation, self.codec_name, exc_type.__name__, msg)
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2521 with self.assertRaisesRegex(exc_type, full_msg) as caught:
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2522 yield caught
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2523 self.assertIsInstance(caught.exception.__cause__, exc_type)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2524
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2525 def raise_obj(self, args, *kwds):
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2526 # Helper to dynamically change the object raised by a test codec
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2527 raise self.obj_to_raise
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2528
87267 04e1f701aeaa Also chain codec exceptions that allow weakrefs Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	2529 def check_wrapped(self, obj_to_raise, msg, exc_type=RuntimeError):
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2530 self.obj_to_raise = obj_to_raise
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2531 self.set_codec(self.raise_obj, self.raise_obj)
87267 04e1f701aeaa Also chain codec exceptions that allow weakrefs Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	2532 with self.assertWrapped("encoding", exc_type, msg):
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2533 "str_input".encode(self.codec_name)
87267 04e1f701aeaa Also chain codec exceptions that allow weakrefs Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	2534 with self.assertWrapped("encoding", exc_type, msg):
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2535 codecs.encode("str_input", self.codec_name)
87267 04e1f701aeaa Also chain codec exceptions that allow weakrefs Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	2536 with self.assertWrapped("decoding", exc_type, msg):
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2537 b"bytes input".decode(self.codec_name)
87267 04e1f701aeaa Also chain codec exceptions that allow weakrefs Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	2538 with self.assertWrapped("decoding", exc_type, msg):
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2539 codecs.decode(b"bytes input", self.codec_name)
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2540
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2541 def test_raise_by_type(self):
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2542 self.check_wrapped(RuntimeError, "")
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2543
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2544 def test_raise_by_value(self):
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2545 msg = "This should be wrapped"
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2546 self.check_wrapped(RuntimeError(msg), msg)
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2547
87267 04e1f701aeaa Also chain codec exceptions that allow weakrefs Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	2548 def test_raise_grandchild_subclass_exact_size(self):
04e1f701aeaa Also chain codec exceptions that allow weakrefs Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	2549 msg = "This should be wrapped"
04e1f701aeaa Also chain codec exceptions that allow weakrefs Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	2550 class MyRuntimeError(RuntimeError):
04e1f701aeaa Also chain codec exceptions that allow weakrefs Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	2551 __slots__ = ()
04e1f701aeaa Also chain codec exceptions that allow weakrefs Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	2552 self.check_wrapped(MyRuntimeError(msg), msg, MyRuntimeError)
04e1f701aeaa Also chain codec exceptions that allow weakrefs Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	2553
04e1f701aeaa Also chain codec exceptions that allow weakrefs Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	2554 def test_raise_subclass_with_weakref_support(self):
04e1f701aeaa Also chain codec exceptions that allow weakrefs Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	2555 msg = "This should be wrapped"
04e1f701aeaa Also chain codec exceptions that allow weakrefs Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	2556 class MyRuntimeError(RuntimeError):
04e1f701aeaa Also chain codec exceptions that allow weakrefs Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	2557 pass
04e1f701aeaa Also chain codec exceptions that allow weakrefs Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	2558 self.check_wrapped(MyRuntimeError(msg), msg, MyRuntimeError)
04e1f701aeaa Also chain codec exceptions that allow weakrefs Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	2559
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2560 def check_not_wrapped(self, obj_to_raise, msg):
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2561 def raise_obj(args, *kwds):
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2562 raise obj_to_raise
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2563 self.set_codec(raise_obj, raise_obj)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2564 with self.assertRaisesRegex(RuntimeError, msg):
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2565 "str input".encode(self.codec_name)
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2566 with self.assertRaisesRegex(RuntimeError, msg):
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2567 codecs.encode("str input", self.codec_name)
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2568 with self.assertRaisesRegex(RuntimeError, msg):
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2569 b"bytes input".decode(self.codec_name)
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2570 with self.assertRaisesRegex(RuntimeError, msg):
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2571 codecs.decode(b"bytes input", self.codec_name)
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2572
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2573 def test_init_override_is_not_wrapped(self):
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2574 class CustomInit(RuntimeError):
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2575 def __init__(self):
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2576 pass
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2577 self.check_not_wrapped(CustomInit, "")
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2578
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2579 def test_new_override_is_not_wrapped(self):
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2580 class CustomNew(RuntimeError):
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2581 def __new__(cls):
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2582 return super().__new__(cls)
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2583 self.check_not_wrapped(CustomNew, "")
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2584
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2585 def test_instance_attribute_is_not_wrapped(self):
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2586 msg = "This should NOT be wrapped"
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2587 exc = RuntimeError(msg)
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2588 exc.attr = 1
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2589 self.check_not_wrapped(exc, "^{}$".format(msg))
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2590
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2591 def test_non_str_arg_is_not_wrapped(self):
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2592 self.check_not_wrapped(RuntimeError(1), "1")
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2593
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86487 diff changeset	2594 def test_multiple_args_is_not_wrapped(self):
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2595 msg_re = r"^$'a', 'b', 'c'$$"
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2596 self.check_not_wrapped(RuntimeError('a', 'b', 'c'), msg_re)
87109 4ea622c085ca Close 19609: narrow scope of codec exc chaining Nick Coghlan <ncoghlan@gmail.com> parents: 87084 diff changeset	2597
4ea622c085ca Close 19609: narrow scope of codec exc chaining Nick Coghlan <ncoghlan@gmail.com> parents: 87084 diff changeset	2598 # http://bugs.python.org/issue19609
4ea622c085ca Close 19609: narrow scope of codec exc chaining Nick Coghlan <ncoghlan@gmail.com> parents: 87084 diff changeset	2599 def test_codec_lookup_failure_not_wrapped(self):
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2600 msg = "^unknown encoding: {}$".format(self.codec_name)
87109 4ea622c085ca Close 19609: narrow scope of codec exc chaining Nick Coghlan <ncoghlan@gmail.com> parents: 87084 diff changeset	2601 # The initial codec lookup should not be wrapped
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2602 with self.assertRaisesRegex(LookupError, msg):
87109 4ea622c085ca Close 19609: narrow scope of codec exc chaining Nick Coghlan <ncoghlan@gmail.com> parents: 87084 diff changeset	2603 "str input".encode(self.codec_name)
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2604 with self.assertRaisesRegex(LookupError, msg):
87109 4ea622c085ca Close 19609: narrow scope of codec exc chaining Nick Coghlan <ncoghlan@gmail.com> parents: 87084 diff changeset	2605 codecs.encode("str input", self.codec_name)
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2606 with self.assertRaisesRegex(LookupError, msg):
87109 4ea622c085ca Close 19609: narrow scope of codec exc chaining Nick Coghlan <ncoghlan@gmail.com> parents: 87084 diff changeset	2607 b"bytes input".decode(self.codec_name)
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2608 with self.assertRaisesRegex(LookupError, msg):
87109 4ea622c085ca Close 19609: narrow scope of codec exc chaining Nick Coghlan <ncoghlan@gmail.com> parents: 87084 diff changeset	2609 codecs.decode(b"bytes input", self.codec_name)
4ea622c085ca Close 19609: narrow scope of codec exc chaining Nick Coghlan <ncoghlan@gmail.com> parents: 87084 diff changeset	2610
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2611 def test_unflagged_non_text_codec_handling(self):
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2612 # The stdlib non-text codecs are now marked so they're
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2613 # pre-emptively skipped by the text model related methods
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2614 # However, third party codecs won't be flagged, so we still make
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2615 # sure the case where an inappropriate output type is produced is
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2616 # handled appropriately
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2617 def encode_to_str(args, *kwds):
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2618 return "not bytes!", 0
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2619 def decode_to_bytes(args, *kwds):
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2620 return b"not str!", 0
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2621 self.set_codec(encode_to_str, decode_to_bytes)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2622 # No input or output type checks on the codecs module functions
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2623 encoded = codecs.encode(None, self.codec_name)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2624 self.assertEqual(encoded, "not bytes!")
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2625 decoded = codecs.decode(None, self.codec_name)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2626 self.assertEqual(decoded, b"not str!")
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2627 # Text model methods should complain
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2628 fmt = (r"^{!r} encoder returned 'str' instead of 'bytes'; "
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2629 "use codecs.encode to encode to arbitrary types$")
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2630 msg = fmt.format(self.codec_name)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2631 with self.assertRaisesRegex(TypeError, msg):
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2632 "str_input".encode(self.codec_name)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2633 fmt = (r"^{!r} decoder returned 'bytes' instead of 'str'; "
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2634 "use codecs.decode to decode to arbitrary types$")
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2635 msg = fmt.format(self.codec_name)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2636 with self.assertRaisesRegex(TypeError, msg):
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2637 b"bytes input".decode(self.codec_name)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87267 diff changeset	2638
85930 d90f25e1a705 Close #17839: support bytes-like objects in base64 module Nick Coghlan <ncoghlan@gmail.com> parents: 81820 diff changeset	2639
66605 7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2640
72977 5841920d1ef6 Issue #12281: Skip code page tests on non-Windows platforms Victor Stinner <victor.stinner@haypocalc.com> parents: 72976 diff changeset	2641 @unittest.skipUnless(sys.platform == 'win32',
5841920d1ef6 Issue #12281: Skip code page tests on non-Windows platforms Victor Stinner <victor.stinner@haypocalc.com> parents: 72976 diff changeset	2642 'code pages are specific to Windows')
72976 af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2643 class CodePageTest(unittest.TestCase):
73138 2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	2644 # CP_UTF8 is already tested by CP65001Test
72976 af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2645 CP_UTF8 = 65001
72977 5841920d1ef6 Issue #12281: Skip code page tests on non-Windows platforms Victor Stinner <victor.stinner@haypocalc.com> parents: 72976 diff changeset	2646
72976 af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2647 def test_invalid_code_page(self):
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2648 self.assertRaises(ValueError, codecs.code_page_encode, -1, 'a')
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2649 self.assertRaises(ValueError, codecs.code_page_decode, -1, b'a')
80944 bb94f6222fef Issue #16719: Get rid of WindowsError. Use OSError instead Andrew Svetlov <andrew.svetlov@gmail.com> parents: 80217 diff changeset	2650 self.assertRaises(OSError, codecs.code_page_encode, 123, 'a')
bb94f6222fef Issue #16719: Get rid of WindowsError. Use OSError instead Andrew Svetlov <andrew.svetlov@gmail.com> parents: 80217 diff changeset	2651 self.assertRaises(OSError, codecs.code_page_decode, 123, b'a')
72976 af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2652
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2653 def test_code_page_name(self):
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2654 self.assertRaisesRegex(UnicodeEncodeError, 'cp932',
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2655 codecs.code_page_encode, 932, '\xff')
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2656 self.assertRaisesRegex(UnicodeDecodeError, 'cp932',
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2657 codecs.code_page_decode, 932, b'\x81\x00')
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2658 self.assertRaisesRegex(UnicodeDecodeError, 'CP_UTF8',
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2659 codecs.code_page_decode, self.CP_UTF8, b'\xff')
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2660
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2661 def check_decode(self, cp, tests):
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2662 for raw, errors, expected in tests:
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2663 if expected is not None:
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2664 try:
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2665 decoded = codecs.code_page_decode(cp, raw, errors)
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2666 except UnicodeDecodeError as err:
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2667 self.fail('Unable to decode %a from "cp%s" with '
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2668 'errors=%r: %s' % (raw, cp, errors, err))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2669 self.assertEqual(decoded[0], expected,
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2670 '%a.decode("cp%s", %r)=%a != %a'
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2671 % (raw, cp, errors, decoded[0], expected))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2672 # assert 0 <= decoded[1] <= len(raw)
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2673 self.assertGreaterEqual(decoded[1], 0)
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2674 self.assertLessEqual(decoded[1], len(raw))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2675 else:
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2676 self.assertRaises(UnicodeDecodeError,
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2677 codecs.code_page_decode, cp, raw, errors)
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2678
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2679 def check_encode(self, cp, tests):
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2680 for text, errors, expected in tests:
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2681 if expected is not None:
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2682 try:
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2683 encoded = codecs.code_page_encode(cp, text, errors)
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2684 except UnicodeEncodeError as err:
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2685 self.fail('Unable to encode %a to "cp%s" with '
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2686 'errors=%r: %s' % (text, cp, errors, err))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2687 self.assertEqual(encoded[0], expected,
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2688 '%a.encode("cp%s", %r)=%a != %a'
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2689 % (text, cp, errors, encoded[0], expected))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2690 self.assertEqual(encoded[1], len(text))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2691 else:
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2692 self.assertRaises(UnicodeEncodeError,
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2693 codecs.code_page_encode, cp, text, errors)
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2694
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2695 def test_cp932(self):
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2696 self.check_encode(932, (
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2697 ('abc', 'strict', b'abc'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2698 ('\uff44\u9a3e', 'strict', b'\x82\x84\xe9\x80'),
73138 2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	2699 # test error handlers
72976 af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2700 ('\xff', 'strict', None),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2701 ('[\xff]', 'ignore', b'[]'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2702 ('[\xff]', 'replace', b'[y]'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2703 ('[\u20ac]', 'replace', b'[?]'),
73138 2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	2704 ('[\xff]', 'backslashreplace', b'[\\xff]'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	2705 ('[\xff]', 'xmlcharrefreplace', b'[ÿ]'),
72976 af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2706 ))
72978 413b89242766 Issue #12281: Fix test_codecs.test_cp932() on Windows XP Victor Stinner <victor.stinner@haypocalc.com> parents: 72977 diff changeset	2707 self.check_decode(932, (
72976 af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2708 (b'abc', 'strict', 'abc'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2709 (b'\x82\x84\xe9\x80', 'strict', '\uff44\u9a3e'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2710 # invalid bytes
73138 2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	2711 (b'[\xff]', 'strict', None),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	2712 (b'[\xff]', 'ignore', '[]'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	2713 (b'[\xff]', 'replace', '[\ufffd]'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	2714 (b'[\xff]', 'surrogateescape', '[\udcff]'),
72976 af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2715 (b'\x81\x00abc', 'strict', None),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2716 (b'\x81\x00abc', 'ignore', '\x00abc'),
72978 413b89242766 Issue #12281: Fix test_codecs.test_cp932() on Windows XP Victor Stinner <victor.stinner@haypocalc.com> parents: 72977 diff changeset	2717 (b'\x81\x00abc', 'replace', '\ufffd\x00abc'),
413b89242766 Issue #12281: Fix test_codecs.test_cp932() on Windows XP Victor Stinner <victor.stinner@haypocalc.com> parents: 72977 diff changeset	2718 ))
72976 af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2719
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2720 def test_cp1252(self):
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2721 self.check_encode(1252, (
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2722 ('abc', 'strict', b'abc'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2723 ('\xe9\u20ac', 'strict', b'\xe9\x80'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2724 ('\xff', 'strict', b'\xff'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2725 ('\u0141', 'strict', None),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2726 ('\u0141', 'ignore', b''),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2727 ('\u0141', 'replace', b'L'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2728 ))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2729 self.check_decode(1252, (
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2730 (b'abc', 'strict', 'abc'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2731 (b'\xe9\x80', 'strict', '\xe9\u20ac'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2732 (b'\xff', 'strict', '\xff'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2733 ))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2734
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2735 def test_cp_utf7(self):
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2736 cp = 65000
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2737 self.check_encode(cp, (
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2738 ('abc', 'strict', b'abc'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2739 ('\xe9\u20ac', 'strict', b'+AOkgrA-'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2740 ('\U0010ffff', 'strict', b'+2//f/w-'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2741 ('\udc80', 'strict', b'+3IA-'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2742 ('\ufffd', 'strict', b'+//0-'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2743 ))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2744 self.check_decode(cp, (
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2745 (b'abc', 'strict', 'abc'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2746 (b'+AOkgrA-', 'strict', '\xe9\u20ac'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2747 (b'+2//f/w-', 'strict', '\U0010ffff'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2748 (b'+3IA-', 'strict', '\udc80'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2749 (b'+//0-', 'strict', '\ufffd'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2750 # invalid bytes
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2751 (b'[+/]', 'strict', '[]'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2752 (b'[\xff]', 'strict', '[\xff]'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2753 ))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2754
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2755 def test_multibyte_encoding(self):
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2756 self.check_decode(932, (
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2757 (b'\x84\xe9\x80', 'ignore', '\u9a3e'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2758 (b'\x84\xe9\x80', 'replace', '\ufffd\u9a3e'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2759 ))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2760 self.check_decode(self.CP_UTF8, (
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2761 (b'\xff\xf4\x8f\xbf\xbf', 'ignore', '\U0010ffff'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2762 (b'\xff\xf4\x8f\xbf\xbf', 'replace', '\ufffd\U0010ffff'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2763 ))
73138 2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	2764 if VISTA_OR_LATER:
72976 af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2765 self.check_encode(self.CP_UTF8, (
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2766 ('[\U0010ffff\uDC80]', 'ignore', b'[\xf4\x8f\xbf\xbf]'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2767 ('[\U0010ffff\uDC80]', 'replace', b'[\xf4\x8f\xbf\xbf?]'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2768 ))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2769
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2770 def test_incremental(self):
73336 17341b93871b Cleanup decode_code_page_stateful() and encode_code_page() Victor Stinner <victor.stinner@haypocalc.com> parents: 73138 diff changeset	2771 decoded = codecs.code_page_decode(932, b'\x82', 'strict', False)
17341b93871b Cleanup decode_code_page_stateful() and encode_code_page() Victor Stinner <victor.stinner@haypocalc.com> parents: 73138 diff changeset	2772 self.assertEqual(decoded, ('', 0))
17341b93871b Cleanup decode_code_page_stateful() and encode_code_page() Victor Stinner <victor.stinner@haypocalc.com> parents: 73138 diff changeset	2773
72976 af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2774 decoded = codecs.code_page_decode(932,
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2775 b'\xe9\x80\xe9', 'strict',
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2776 False)
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2777 self.assertEqual(decoded, ('\u9a3e', 2))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2778
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2779 decoded = codecs.code_page_decode(932,
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2780 b'\xe9\x80\xe9\x80', 'strict',
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2781 False)
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2782 self.assertEqual(decoded, ('\u9a3e\u9a3e', 4))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2783
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2784 decoded = codecs.code_page_decode(932,
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2785 b'abc', 'strict',
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2786 False)
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2787 self.assertEqual(decoded, ('abc', 3))
20013 e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	2788
e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	2789
e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	2790 if __name__ == "__main__":
81383 11e7d0936fa2 #16918: test_codecs now works with unittest test discovery. Patch by Zachary Ware. Ezio Melotti <ezio.melotti@gmail.com> parents: 81382 diff changeset	2791 unittest.main()

Mercurial > cpython-lru_cache-weakref

annotate Lib/test/test_codecs.py @ 87690:66c1c9f32567 default tip