cpython-lru_cache-weakref: Lib/test/test

annotate Lib/test/test_codecs.py @ 79691:05c8928362ee

Merge __next__ method link fixes with 3.3.

author	Ezio Melotti <ezio.melotti@gmail.com>
date	Fri, 12 Oct 2012 13:46:37 +0300
parents	c64dec45d46f
children	fd41c6b4de57

rev	line source
73579 ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	1 import _testcapi
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	2 import codecs
73579 ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	3 import io
71489 92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70293 diff changeset	4 import locale
73579 ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	5 import sys
ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	6 import unittest
ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	7 import warnings
ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	8
ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	9 from test import support
72534 85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	10
73138 2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	11 if sys.platform == 'win32':
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	12 VISTA_OR_LATER = (sys.getwindowsversion().major >= 6)
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	13 else:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	14 VISTA_OR_LATER = False
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	15
72683 da3bfa2e9daa Fix text failures when ctypes is not available Antoine Pitrou <solipsis@pitrou.net> parents: 72534 diff changeset	16 try:
da3bfa2e9daa Fix text failures when ctypes is not available Antoine Pitrou <solipsis@pitrou.net> parents: 72534 diff changeset	17 import ctypes
da3bfa2e9daa Fix text failures when ctypes is not available Antoine Pitrou <solipsis@pitrou.net> parents: 72534 diff changeset	18 except ImportError:
da3bfa2e9daa Fix text failures when ctypes is not available Antoine Pitrou <solipsis@pitrou.net> parents: 72534 diff changeset	19 ctypes = None
da3bfa2e9daa Fix text failures when ctypes is not available Antoine Pitrou <solipsis@pitrou.net> parents: 72534 diff changeset	20 SIZEOF_WCHAR_T = -1
da3bfa2e9daa Fix text failures when ctypes is not available Antoine Pitrou <solipsis@pitrou.net> parents: 72534 diff changeset	21 else:
da3bfa2e9daa Fix text failures when ctypes is not available Antoine Pitrou <solipsis@pitrou.net> parents: 72534 diff changeset	22 SIZEOF_WCHAR_T = ctypes.sizeof(ctypes.c_wchar)
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	23
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	24 class Queue(object):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	25 """
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	26 queue: write bytes at one end, read bytes from the other end
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	27 """
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	28 def __init__(self, buffer):
a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	29 self._buffer = buffer
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	30
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	31 def write(self, chars):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	32 self._buffer += chars
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	33
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	34 def read(self, size=-1):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	35 if size<0:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	36 s = self._buffer
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	37 self._buffer = self._buffer[:0] # make empty
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	38 return s
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	39 else:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	40 s = self._buffer[:size]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	41 self._buffer = self._buffer[size:]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	42 return s
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	43
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	44 class MixInCheckStateHandling:
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	45 def check_state_handling_decode(self, encoding, u, s):
41642 a6eb6acfe04a Merged revisions 55007-55179 via svnmerge from Guido van Rossum <guido@python.org> parents: 41606 diff changeset	46 for i in range(len(s)+1):
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	47 d = codecs.getincrementaldecoder(encoding)()
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	48 part1 = d.decode(s[:i])
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	49 state = d.getstate()
58473 71acd4482e65 Merged revisions 77727 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 56036 diff changeset	50 self.assertIsInstance(state[1], int)
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	51 # Check that the condition stated in the documentation for
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	52 # IncrementalDecoder.getstate() holds
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	53 if not state[1]:
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	54 # reset decoder to the default state without anything buffered
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	55 d.setstate((state[0][:0], 0))
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	56 # Feeding the previous input may not produce any output
55593 0bb232078b8c Merged revisions 73715 via svnmerge from Georg Brandl <georg@python.org> parents: 54988 diff changeset	57 self.assertTrue(not d.decode(state[0]))
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	58 # The decoder must return to the same state
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	59 self.assertEqual(state, d.getstate())
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	60 # Create a new decoder and set it to the state
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	61 # we extracted from the old one
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	62 d = codecs.getincrementaldecoder(encoding)()
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	63 d.setstate(state)
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	64 part2 = d.decode(s[i:], True)
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	65 self.assertEqual(u, part1+part2)
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	66
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	67 def check_state_handling_encode(self, encoding, u, s):
41642 a6eb6acfe04a Merged revisions 55007-55179 via svnmerge from Guido van Rossum <guido@python.org> parents: 41606 diff changeset	68 for i in range(len(u)+1):
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	69 d = codecs.getincrementalencoder(encoding)()
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	70 part1 = d.encode(u[:i])
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	71 state = d.getstate()
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	72 d = codecs.getincrementalencoder(encoding)()
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	73 d.setstate(state)
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	74 part2 = d.encode(u[i:], True)
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	75 self.assertEqual(s, part1+part2)
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	76
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	77 class ReadTest(unittest.TestCase, MixInCheckStateHandling):
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	78 def check_partial(self, input, partialresults):
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	79 # get a StreamReader for the encoding and feed the bytestring version
43912 4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	80 # of input to the reader byte by byte. Read everything available from
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	81 # the StreamReader and check that the results equal the appropriate
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	82 # entries from partialresults.
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	83 q = Queue(b"")
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	84 r = codecs.getreader(self.encoding)(q)
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	85 result = ""
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	86 for (c, partialresult) in zip(input.encode(self.encoding), partialresults):
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	87 q.write(bytes([c]))
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	88 result += r.read()
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	89 self.assertEqual(result, partialresult)
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	90 # check that there's nothing left in the buffers
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	91 self.assertEqual(r.read(), "")
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	92 self.assertEqual(r.bytebuffer, b"")
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	93
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	94 # do the check again, this time using a incremental decoder
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	95 d = codecs.getincrementaldecoder(self.encoding)()
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	96 result = ""
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	97 for (c, partialresult) in zip(input.encode(self.encoding), partialresults):
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	98 result += d.decode(bytes([c]))
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	99 self.assertEqual(result, partialresult)
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	100 # check that there's nothing left in the buffers
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	101 self.assertEqual(d.decode(b"", True), "")
a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	102 self.assertEqual(d.buffer, b"")
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	103
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	104 # Check whether the reset method works properly
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	105 d.reset()
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	106 result = ""
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	107 for (c, partialresult) in zip(input.encode(self.encoding), partialresults):
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	108 result += d.decode(bytes([c]))
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	109 self.assertEqual(result, partialresult)
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	110 # check that there's nothing left in the buffers
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	111 self.assertEqual(d.decode(b"", True), "")
a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	112 self.assertEqual(d.buffer, b"")
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	113
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	114 # check iterdecode()
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	115 encoded = input.encode(self.encoding)
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	116 self.assertEqual(
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	117 input,
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	118 "".join(codecs.iterdecode([bytes([c]) for c in encoded], self.encoding))
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	119 )
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	120
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	121 def test_readline(self):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	122 def getreader(input):
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	123 stream = io.BytesIO(input.encode(self.encoding))
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	124 return codecs.getreader(self.encoding)(stream)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	125
36855 204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	126 def readalllines(input, keepends=True, size=None):
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	127 reader = getreader(input)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	128 lines = []
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	129 while True:
36855 204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	130 line = reader.readline(size=size, keepends=keepends)
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	131 if not line:
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	132 break
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	133 lines.append(line)
36855 204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	134 return "\|".join(lines)
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	135
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	136 s = "foo\nbar\r\nbaz\rspam\u2028eggs"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	137 sexpected = "foo\n\|bar\r\n\|baz\r\|spam\u2028\|eggs"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	138 sexpectednoends = "foo\|bar\|baz\|spam\|eggs"
36855 204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	139 self.assertEqual(readalllines(s, True), sexpected)
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	140 self.assertEqual(readalllines(s, False), sexpectednoends)
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	141 self.assertEqual(readalllines(s, True, 10), sexpected)
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	142 self.assertEqual(readalllines(s, False, 10), sexpectednoends)
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	143
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	144 # Test long lines (multiple calls to read() in readline())
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	145 vw = []
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	146 vwo = []
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	147 for (i, lineend) in enumerate("\n \r\n \r \u2028".split()):
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	148 vw.append((i200)"\3042" + lineend)
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	149 vwo.append((i200)"\3042")
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	150 self.assertEqual(readalllines("".join(vw), True), "".join(vw))
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	151 self.assertEqual(readalllines("".join(vw), False),"".join(vwo))
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	152
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	153 # Test lines where the first read might end with \r, so the
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	154 # reader has to look ahead whether this is a lone \r or a \r\n
41642 a6eb6acfe04a Merged revisions 55007-55179 via svnmerge from Guido van Rossum <guido@python.org> parents: 41606 diff changeset	155 for size in range(80):
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	156 for lineend in "\n \r\n \r \u2028".split():
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	157 s = 10(size"a" + lineend + "xxx\n")
34865 b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	158 reader = getreader(s)
41642 a6eb6acfe04a Merged revisions 55007-55179 via svnmerge from Guido van Rossum <guido@python.org> parents: 41606 diff changeset	159 for i in range(10):
34865 b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	160 self.assertEqual(
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	161 reader.readline(keepends=True),
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	162 size*"a" + lineend,
34865 b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	163 )
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	164 reader = getreader(s)
41642 a6eb6acfe04a Merged revisions 55007-55179 via svnmerge from Guido van Rossum <guido@python.org> parents: 41606 diff changeset	165 for i in range(10):
34865 b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	166 self.assertEqual(
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	167 reader.readline(keepends=False),
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	168 size*"a",
34865 b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	169 )
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	170
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	171 def test_bug1175396(self):
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	172 s = [
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	173 '<%!--===================================================\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	174 ' BLOG index page: show recent articles,\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	175 ' today\'s articles, or articles of a specific date.\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	176 '========================================================--%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	177 '<%@inputencoding="ISO-8859-1"%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	178 '<%@pagetemplate=TEMPLATE.y%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	179 '<%@import=import frog.util, frog%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	180 '<%@import=import frog.objects%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	181 '<%@import=from frog.storageerrors import StorageError%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	182 '<%\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	183 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	184 'import logging\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	185 'log=logging.getLogger("Snakelets.logger")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	186 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	187 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	188 'user=self.SessionCtx.user\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	189 'storageEngine=self.SessionCtx.storageEngine\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	190 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	191 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	192 'def readArticlesFromDate(date, count=None):\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	193 ' entryids=storageEngine.listBlogEntries(date)\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	194 ' entryids.reverse() # descending\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	195 ' if count:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	196 ' entryids=entryids[:count]\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	197 ' try:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	198 ' return [ frog.objects.BlogEntry.load(storageEngine, date, Id) for Id in entryids ]\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	199 ' except StorageError,x:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	200 ' log.error("Error loading articles: "+str(x))\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	201 ' self.abort("cannot load articles")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	202 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	203 'showdate=None\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	204 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	205 'arg=self.Request.getArg()\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	206 'if arg=="today":\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	207 ' #-------------------- TODAY\'S ARTICLES\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	208 ' self.write("<h2>Today\'s articles</h2>")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	209 ' showdate = frog.util.isodatestr() \r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	210 ' entries = readArticlesFromDate(showdate)\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	211 'elif arg=="active":\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	212 ' #-------------------- ACTIVE ARTICLES redirect\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	213 ' self.Yredirect("active.y")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	214 'elif arg=="login":\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	215 ' #-------------------- LOGIN PAGE redirect\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	216 ' self.Yredirect("login.y")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	217 'elif arg=="date":\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	218 ' #-------------------- ARTICLES OF A SPECIFIC DATE\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	219 ' showdate = self.Request.getParameter("date")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	220 ' self.write("<h2>Articles written on %s</h2>"% frog.util.mediumdatestr(showdate))\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	221 ' entries = readArticlesFromDate(showdate)\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	222 'else:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	223 ' #-------------------- RECENT ARTICLES\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	224 ' self.write("<h2>Recent articles</h2>")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	225 ' dates=storageEngine.listBlogEntryDates()\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	226 ' if dates:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	227 ' entries=[]\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	228 ' SHOWAMOUNT=10\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	229 ' for showdate in dates:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	230 ' entries.extend( readArticlesFromDate(showdate, SHOWAMOUNT-len(entries)) )\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	231 ' if len(entries)>=SHOWAMOUNT:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	232 ' break\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	233 ' \r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	234 ]
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	235 stream = io.BytesIO("".join(s).encode(self.encoding))
34865 b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	236 reader = codecs.getreader(self.encoding)(stream)
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	237 for (i, line) in enumerate(reader):
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	238 self.assertEqual(line, s[i])
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	239
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	240 def test_readlinequeue(self):
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	241 q = Queue(b"")
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	242 writer = codecs.getwriter(self.encoding)(q)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	243 reader = codecs.getreader(self.encoding)(q)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	244
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	245 # No lineends
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	246 writer.write("foo\r")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	247 self.assertEqual(reader.readline(keepends=False), "foo")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	248 writer.write("\nbar\r")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	249 self.assertEqual(reader.readline(keepends=False), "")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	250 self.assertEqual(reader.readline(keepends=False), "bar")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	251 writer.write("baz")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	252 self.assertEqual(reader.readline(keepends=False), "baz")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	253 self.assertEqual(reader.readline(keepends=False), "")
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	254
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	255 # Lineends
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	256 writer.write("foo\r")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	257 self.assertEqual(reader.readline(keepends=True), "foo\r")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	258 writer.write("\nbar\r")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	259 self.assertEqual(reader.readline(keepends=True), "\n")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	260 self.assertEqual(reader.readline(keepends=True), "bar\r")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	261 writer.write("baz")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	262 self.assertEqual(reader.readline(keepends=True), "baz")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	263 self.assertEqual(reader.readline(keepends=True), "")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	264 writer.write("foo\r\n")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	265 self.assertEqual(reader.readline(keepends=True), "foo\r\n")
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	266
34384 a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	267 def test_bug1098990_a(self):
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	268 s1 = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy\r\n"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	269 s2 = "offending line: ladfj askldfj klasdj fskla dfzaskdj fasklfj laskd fjasklfzzzzaa%whereisthis!!!\r\n"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	270 s3 = "next line.\r\n"
34384 a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	271
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	272 s = (s1+s2+s3).encode(self.encoding)
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	273 stream = io.BytesIO(s)
34384 a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	274 reader = codecs.getreader(self.encoding)(stream)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	275 self.assertEqual(reader.readline(), s1)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	276 self.assertEqual(reader.readline(), s2)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	277 self.assertEqual(reader.readline(), s3)
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	278 self.assertEqual(reader.readline(), "")
34384 a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	279
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	280 def test_bug1098990_b(self):
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	281 s1 = "aaaaaaaaaaaaaaaaaaaaaaaa\r\n"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	282 s2 = "bbbbbbbbbbbbbbbbbbbbbbbb\r\n"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	283 s3 = "stillokay:bbbbxx\r\n"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	284 s4 = "broken!!!!badbad\r\n"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	285 s5 = "againokay.\r\n"
34384 a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	286
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	287 s = (s1+s2+s3+s4+s5).encode(self.encoding)
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	288 stream = io.BytesIO(s)
34384 a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	289 reader = codecs.getreader(self.encoding)(stream)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	290 self.assertEqual(reader.readline(), s1)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	291 self.assertEqual(reader.readline(), s2)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	292 self.assertEqual(reader.readline(), s3)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	293 self.assertEqual(reader.readline(), s4)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	294 self.assertEqual(reader.readline(), s5)
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	295 self.assertEqual(reader.readline(), "")
34384 a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	296
42626 f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	297 class UTF32Test(ReadTest):
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	298 encoding = "utf-32"
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	299
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	300 spamle = (b'\xff\xfe\x00\x00'
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	301 b's\x00\x00\x00p\x00\x00\x00a\x00\x00\x00m\x00\x00\x00'
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	302 b's\x00\x00\x00p\x00\x00\x00a\x00\x00\x00m\x00\x00\x00')
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	303 spambe = (b'\x00\x00\xfe\xff'
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	304 b'\x00\x00\x00s\x00\x00\x00p\x00\x00\x00a\x00\x00\x00m'
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	305 b'\x00\x00\x00s\x00\x00\x00p\x00\x00\x00a\x00\x00\x00m')
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	306
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	307 def test_only_one_bom(self):
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	308 _,_,reader,writer = codecs.lookup(self.encoding)
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	309 # encode some stream
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	310 s = io.BytesIO()
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	311 f = writer(s)
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	312 f.write("spam")
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	313 f.write("spam")
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	314 d = s.getvalue()
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	315 # check whether there is exactly one BOM in it
55593 0bb232078b8c Merged revisions 73715 via svnmerge from Georg Brandl <georg@python.org> parents: 54988 diff changeset	316 self.assertTrue(d == self.spamle or d == self.spambe)
42626 f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	317 # try to read it back
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	318 s = io.BytesIO(d)
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	319 f = reader(s)
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	320 self.assertEqual(f.read(), "spamspam")
42626 f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	321
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	322 def test_badbom(self):
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	323 s = io.BytesIO(4*b"\xff")
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	324 f = codecs.getreader(self.encoding)(s)
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	325 self.assertRaises(UnicodeError, f.read)
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	326
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	327 s = io.BytesIO(8*b"\xff")
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	328 f = codecs.getreader(self.encoding)(s)
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	329 self.assertRaises(UnicodeError, f.read)
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	330
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	331 def test_partial(self):
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	332 self.check_partial(
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	333 "\x00\xff\u0100\uffff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	334 [
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	335 "", # first byte of BOM read
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	336 "", # second byte of BOM read
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	337 "", # third byte of BOM read
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	338 "", # fourth byte of BOM read => byteorder known
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	339 "",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	340 "",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	341 "",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	342 "\x00",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	343 "\x00",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	344 "\x00",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	345 "\x00",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	346 "\x00\xff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	347 "\x00\xff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	348 "\x00\xff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	349 "\x00\xff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	350 "\x00\xff\u0100",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	351 "\x00\xff\u0100",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	352 "\x00\xff\u0100",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	353 "\x00\xff\u0100",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	354 "\x00\xff\u0100\uffff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	355 ]
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	356 )
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	357
56036 a9809f3b164f Merged revisions 74871 via svnmerge from Georg Brandl <georg@python.org> parents: 55593 diff changeset	358 def test_handlers(self):
a9809f3b164f Merged revisions 74871 via svnmerge from Georg Brandl <georg@python.org> parents: 55593 diff changeset	359 self.assertEqual(('\ufffd', 1),
a9809f3b164f Merged revisions 74871 via svnmerge from Georg Brandl <georg@python.org> parents: 55593 diff changeset	360 codecs.utf_32_decode(b'\x01', 'replace', True))
a9809f3b164f Merged revisions 74871 via svnmerge from Georg Brandl <georg@python.org> parents: 55593 diff changeset	361 self.assertEqual(('', 1),
a9809f3b164f Merged revisions 74871 via svnmerge from Georg Brandl <georg@python.org> parents: 55593 diff changeset	362 codecs.utf_32_decode(b'\x01', 'ignore', True))
a9809f3b164f Merged revisions 74871 via svnmerge from Georg Brandl <georg@python.org> parents: 55593 diff changeset	363
42626 f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	364 def test_errors(self):
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	365 self.assertRaises(UnicodeDecodeError, codecs.utf_32_decode,
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	366 b"\xff", "strict", True)
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	367
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	368 def test_decoder_state(self):
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	369 self.check_state_handling_decode(self.encoding,
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	370 "spamspam", self.spamle)
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	371 self.check_state_handling_decode(self.encoding,
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	372 "spamspam", self.spambe)
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	373
62258 8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	374 def test_issue8941(self):
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	375 # Issue #8941: insufficient result allocation when decoding into
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	376 # surrogate pairs on UCS-2 builds.
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	377 encoded_le = b'\xff\xfe\x00\x00' + b'\x00\x00\x01\x00' * 1024
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	378 self.assertEqual('\U00010000' * 1024,
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	379 codecs.utf_32_decode(encoded_le)[0])
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	380 encoded_be = b'\x00\x00\xfe\xff' + b'\x00\x01\x00\x00' * 1024
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	381 self.assertEqual('\U00010000' * 1024,
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	382 codecs.utf_32_decode(encoded_be)[0])
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	383
42626 f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	384 class UTF32LETest(ReadTest):
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	385 encoding = "utf-32-le"
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	386
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	387 def test_partial(self):
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	388 self.check_partial(
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	389 "\x00\xff\u0100\uffff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	390 [
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	391 "",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	392 "",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	393 "",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	394 "\x00",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	395 "\x00",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	396 "\x00",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	397 "\x00",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	398 "\x00\xff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	399 "\x00\xff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	400 "\x00\xff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	401 "\x00\xff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	402 "\x00\xff\u0100",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	403 "\x00\xff\u0100",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	404 "\x00\xff\u0100",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	405 "\x00\xff\u0100",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	406 "\x00\xff\u0100\uffff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	407 ]
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	408 )
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	409
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	410 def test_simple(self):
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	411 self.assertEqual("\U00010203".encode(self.encoding), b"\x03\x02\x01\x00")
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	412
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	413 def test_errors(self):
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	414 self.assertRaises(UnicodeDecodeError, codecs.utf_32_le_decode,
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	415 b"\xff", "strict", True)
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	416
62258 8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	417 def test_issue8941(self):
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	418 # Issue #8941: insufficient result allocation when decoding into
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	419 # surrogate pairs on UCS-2 builds.
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	420 encoded = b'\x00\x00\x01\x00' * 1024
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	421 self.assertEqual('\U00010000' * 1024,
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	422 codecs.utf_32_le_decode(encoded)[0])
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	423
42626 f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	424 class UTF32BETest(ReadTest):
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	425 encoding = "utf-32-be"
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	426
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	427 def test_partial(self):
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	428 self.check_partial(
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	429 "\x00\xff\u0100\uffff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	430 [
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	431 "",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	432 "",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	433 "",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	434 "\x00",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	435 "\x00",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	436 "\x00",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	437 "\x00",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	438 "\x00\xff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	439 "\x00\xff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	440 "\x00\xff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	441 "\x00\xff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	442 "\x00\xff\u0100",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	443 "\x00\xff\u0100",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	444 "\x00\xff\u0100",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	445 "\x00\xff\u0100",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	446 "\x00\xff\u0100\uffff",
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	447 ]
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	448 )
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	449
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	450 def test_simple(self):
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	451 self.assertEqual("\U00010203".encode(self.encoding), b"\x00\x01\x02\x03")
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	452
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	453 def test_errors(self):
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	454 self.assertRaises(UnicodeDecodeError, codecs.utf_32_be_decode,
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	455 b"\xff", "strict", True)
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	456
62258 8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	457 def test_issue8941(self):
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	458 # Issue #8941: insufficient result allocation when decoding into
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	459 # surrogate pairs on UCS-2 builds.
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	460 encoded = b'\x00\x01\x00\x00' * 1024
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	461 self.assertEqual('\U00010000' * 1024,
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	462 codecs.utf_32_be_decode(encoded)[0])
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	463
8d5dd2276680 Merged revisions 81908 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62215 diff changeset	464
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	465 class UTF16Test(ReadTest):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	466 encoding = "utf-16"
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	467
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	468 spamle = b'\xff\xfes\x00p\x00a\x00m\x00s\x00p\x00a\x00m\x00'
a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	469 spambe = b'\xfe\xff\x00s\x00p\x00a\x00m\x00s\x00p\x00a\x00m'
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	470
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	471 def test_only_one_bom(self):
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	472 _,_,reader,writer = codecs.lookup(self.encoding)
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	473 # encode some stream
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	474 s = io.BytesIO()
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	475 f = writer(s)
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	476 f.write("spam")
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	477 f.write("spam")
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	478 d = s.getvalue()
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	479 # check whether there is exactly one BOM in it
55593 0bb232078b8c Merged revisions 73715 via svnmerge from Georg Brandl <georg@python.org> parents: 54988 diff changeset	480 self.assertTrue(d == self.spamle or d == self.spambe)
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	481 # try to read it back
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	482 s = io.BytesIO(d)
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	483 f = reader(s)
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	484 self.assertEqual(f.read(), "spamspam")
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	485
34538 37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	486 def test_badbom(self):
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	487 s = io.BytesIO(b"\xff\xff")
34562 7a1277fad04e Fix stupid typo: Don't read from a writer. Walter Dörwald <walter@livinglogic.de> parents: 34538 diff changeset	488 f = codecs.getreader(self.encoding)(s)
34538 37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	489 self.assertRaises(UnicodeError, f.read)
37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	490
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	491 s = io.BytesIO(b"\xff\xff\xff\xff")
34562 7a1277fad04e Fix stupid typo: Don't read from a writer. Walter Dörwald <walter@livinglogic.de> parents: 34538 diff changeset	492 f = codecs.getreader(self.encoding)(s)
34538 37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	493 self.assertRaises(UnicodeError, f.read)
37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	494
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	495 def test_partial(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	496 self.check_partial(
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	497 "\x00\xff\u0100\uffff",
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	498 [
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	499 "", # first byte of BOM read
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	500 "", # second byte of BOM read => byteorder known
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	501 "",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	502 "\x00",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	503 "\x00",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	504 "\x00\xff",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	505 "\x00\xff",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	506 "\x00\xff\u0100",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	507 "\x00\xff\u0100",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	508 "\x00\xff\u0100\uffff",
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	509 ]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	510 )
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	511
56036 a9809f3b164f Merged revisions 74871 via svnmerge from Georg Brandl <georg@python.org> parents: 55593 diff changeset	512 def test_handlers(self):
a9809f3b164f Merged revisions 74871 via svnmerge from Georg Brandl <georg@python.org> parents: 55593 diff changeset	513 self.assertEqual(('\ufffd', 1),
a9809f3b164f Merged revisions 74871 via svnmerge from Georg Brandl <georg@python.org> parents: 55593 diff changeset	514 codecs.utf_16_decode(b'\x01', 'replace', True))
a9809f3b164f Merged revisions 74871 via svnmerge from Georg Brandl <georg@python.org> parents: 55593 diff changeset	515 self.assertEqual(('', 1),
a9809f3b164f Merged revisions 74871 via svnmerge from Georg Brandl <georg@python.org> parents: 55593 diff changeset	516 codecs.utf_16_decode(b'\x01', 'ignore', True))
a9809f3b164f Merged revisions 74871 via svnmerge from Georg Brandl <georg@python.org> parents: 55593 diff changeset	517
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	518 def test_errors(self):
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	519 self.assertRaises(UnicodeDecodeError, codecs.utf_16_decode,
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	520 b"\xff", "strict", True)
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	521
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	522 def test_decoder_state(self):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	523 self.check_state_handling_decode(self.encoding,
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	524 "spamspam", self.spamle)
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	525 self.check_state_handling_decode(self.encoding,
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	526 "spamspam", self.spambe)
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	527
59171 0d590e5e840c Recorded merge of revisions 78462,78484 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 56036 diff changeset	528 def test_bug691291(self):
0d590e5e840c Recorded merge of revisions 78462,78484 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 56036 diff changeset	529 # Files are always opened in binary mode, even if no binary mode was
0d590e5e840c Recorded merge of revisions 78462,78484 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 56036 diff changeset	530 # specified. This means that no automatic conversion of '\n' is done
0d590e5e840c Recorded merge of revisions 78462,78484 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 56036 diff changeset	531 # on reading and writing.
0d590e5e840c Recorded merge of revisions 78462,78484 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 56036 diff changeset	532 s1 = 'Hello\r\nworld\r\n'
0d590e5e840c Recorded merge of revisions 78462,78484 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 56036 diff changeset	533
0d590e5e840c Recorded merge of revisions 78462,78484 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 56036 diff changeset	534 s = s1.encode(self.encoding)
70292 b538445a9df8 test_codecs now removes the temporay file (created by the test) Victor Stinner <victor.stinner@haypocalc.com> parents: 66340 diff changeset	535 self.addCleanup(support.unlink, support.TESTFN)
b538445a9df8 test_codecs now removes the temporay file (created by the test) Victor Stinner <victor.stinner@haypocalc.com> parents: 66340 diff changeset	536 with open(support.TESTFN, 'wb') as fp:
b538445a9df8 test_codecs now removes the temporay file (created by the test) Victor Stinner <victor.stinner@haypocalc.com> parents: 66340 diff changeset	537 fp.write(s)
b538445a9df8 test_codecs now removes the temporay file (created by the test) Victor Stinner <victor.stinner@haypocalc.com> parents: 66340 diff changeset	538 with codecs.open(support.TESTFN, 'U', encoding=self.encoding) as reader:
b538445a9df8 test_codecs now removes the temporay file (created by the test) Victor Stinner <victor.stinner@haypocalc.com> parents: 66340 diff changeset	539 self.assertEqual(reader.read(), s1)
59171 0d590e5e840c Recorded merge of revisions 78462,78484 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 56036 diff changeset	540
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	541 class UTF16LETest(ReadTest):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	542 encoding = "utf-16-le"
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	543
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	544 def test_partial(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	545 self.check_partial(
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	546 "\x00\xff\u0100\uffff",
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	547 [
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	548 "",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	549 "\x00",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	550 "\x00",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	551 "\x00\xff",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	552 "\x00\xff",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	553 "\x00\xff\u0100",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	554 "\x00\xff\u0100",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	555 "\x00\xff\u0100\uffff",
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	556 ]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	557 )
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	558
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	559 def test_errors(self):
78206 034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	560 tests = [
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	561 (b'\xff', '\ufffd'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	562 (b'A\x00Z', 'A\ufffd'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	563 (b'A\x00B\x00C\x00D\x00Z', 'ABCD\ufffd'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	564 (b'\x00\xd8', '\ufffd'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	565 (b'\x00\xd8A', '\ufffd'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	566 (b'\x00\xd8A\x00', '\ufffdA'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	567 (b'\x00\xdcA\x00', '\ufffdA'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	568 ]
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	569 for raw, expected in tests:
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	570 self.assertRaises(UnicodeDecodeError, codecs.utf_16_le_decode,
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	571 raw, 'strict', True)
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	572 self.assertEqual(raw.decode('utf-16le', 'replace'), expected)
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	573
66797 113925b79b97 Issue #10546: UTF-16-LE and UTF-16-BE do support non-BMP characters Victor Stinner <victor.stinner@haypocalc.com> parents: 66605 diff changeset	574 def test_nonbmp(self):
113925b79b97 Issue #10546: UTF-16-LE and UTF-16-BE do support non-BMP characters Victor Stinner <victor.stinner@haypocalc.com> parents: 66605 diff changeset	575 self.assertEqual("\U00010203".encode(self.encoding),
113925b79b97 Issue #10546: UTF-16-LE and UTF-16-BE do support non-BMP characters Victor Stinner <victor.stinner@haypocalc.com> parents: 66605 diff changeset	576 b'\x00\xd8\x03\xde')
113925b79b97 Issue #10546: UTF-16-LE and UTF-16-BE do support non-BMP characters Victor Stinner <victor.stinner@haypocalc.com> parents: 66605 diff changeset	577 self.assertEqual(b'\x00\xd8\x03\xde'.decode(self.encoding),
113925b79b97 Issue #10546: UTF-16-LE and UTF-16-BE do support non-BMP characters Victor Stinner <victor.stinner@haypocalc.com> parents: 66605 diff changeset	578 "\U00010203")
113925b79b97 Issue #10546: UTF-16-LE and UTF-16-BE do support non-BMP characters Victor Stinner <victor.stinner@haypocalc.com> parents: 66605 diff changeset	579
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	580 class UTF16BETest(ReadTest):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	581 encoding = "utf-16-be"
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	582
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	583 def test_partial(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	584 self.check_partial(
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	585 "\x00\xff\u0100\uffff",
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	586 [
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	587 "",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	588 "\x00",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	589 "\x00",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	590 "\x00\xff",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	591 "\x00\xff",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	592 "\x00\xff\u0100",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	593 "\x00\xff\u0100",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	594 "\x00\xff\u0100\uffff",
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	595 ]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	596 )
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	597
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	598 def test_errors(self):
78206 034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	599 tests = [
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	600 (b'\xff', '\ufffd'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	601 (b'\x00A\xff', 'A\ufffd'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	602 (b'\x00A\x00B\x00C\x00DZ', 'ABCD\ufffd'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	603 (b'\xd8\x00', '\ufffd'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	604 (b'\xd8\x00\xdc', '\ufffd'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	605 (b'\xd8\x00\x00A', '\ufffdA'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	606 (b'\xdc\x00\x00A', '\ufffdA'),
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	607 ]
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	608 for raw, expected in tests:
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	609 self.assertRaises(UnicodeDecodeError, codecs.utf_16_be_decode,
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	610 raw, 'strict', True)
034ff986019d Issue #14579: Fix CVE-2012-2135: vulnerability in the utf-16 decoder after error handling. Antoine Pitrou <solipsis@pitrou.net> parents: 71511 diff changeset	611 self.assertEqual(raw.decode('utf-16be', 'replace'), expected)
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	612
66797 113925b79b97 Issue #10546: UTF-16-LE and UTF-16-BE do support non-BMP characters Victor Stinner <victor.stinner@haypocalc.com> parents: 66605 diff changeset	613 def test_nonbmp(self):
113925b79b97 Issue #10546: UTF-16-LE and UTF-16-BE do support non-BMP characters Victor Stinner <victor.stinner@haypocalc.com> parents: 66605 diff changeset	614 self.assertEqual("\U00010203".encode(self.encoding),
113925b79b97 Issue #10546: UTF-16-LE and UTF-16-BE do support non-BMP characters Victor Stinner <victor.stinner@haypocalc.com> parents: 66605 diff changeset	615 b'\xd8\x00\xde\x03')
113925b79b97 Issue #10546: UTF-16-LE and UTF-16-BE do support non-BMP characters Victor Stinner <victor.stinner@haypocalc.com> parents: 66605 diff changeset	616 self.assertEqual(b'\xd8\x00\xde\x03'.decode(self.encoding),
113925b79b97 Issue #10546: UTF-16-LE and UTF-16-BE do support non-BMP characters Victor Stinner <victor.stinner@haypocalc.com> parents: 66605 diff changeset	617 "\U00010203")
113925b79b97 Issue #10546: UTF-16-LE and UTF-16-BE do support non-BMP characters Victor Stinner <victor.stinner@haypocalc.com> parents: 66605 diff changeset	618
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	619 class UTF8Test(ReadTest):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	620 encoding = "utf-8"
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	621
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	622 def test_partial(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	623 self.check_partial(
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	624 "\x00\xff\u07ff\u0800\uffff",
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	625 [
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	626 "\x00",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	627 "\x00",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	628 "\x00\xff",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	629 "\x00\xff",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	630 "\x00\xff\u07ff",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	631 "\x00\xff\u07ff",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	632 "\x00\xff\u07ff",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	633 "\x00\xff\u07ff\u0800",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	634 "\x00\xff\u07ff\u0800",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	635 "\x00\xff\u07ff\u0800",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	636 "\x00\xff\u07ff\u0800\uffff",
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	637 ]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	638 )
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	639
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	640 def test_decoder_state(self):
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	641 u = "\x00\x7f\x80\xff\u0100\u07ff\u0800\uffff\U0010ffff"
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	642 self.check_state_handling_decode(self.encoding,
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	643 u, u.encode(self.encoding))
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	644
53681 52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 50975 diff changeset	645 def test_lone_surrogates(self):
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 50975 diff changeset	646 self.assertRaises(UnicodeEncodeError, "\ud800".encode, "utf-8")
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 50975 diff changeset	647 self.assertRaises(UnicodeDecodeError, b"\xed\xa0\x80".decode, "utf-8")
60875 149badf73bca Merged revisions 80382 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 59171 diff changeset	648 self.assertEqual("[\uDC80]".encode("utf-8", "backslashreplace"),
149badf73bca Merged revisions 80382 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 59171 diff changeset	649 b'[\\udc80]')
149badf73bca Merged revisions 80382 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 59171 diff changeset	650 self.assertEqual("[\uDC80]".encode("utf-8", "xmlcharrefreplace"),
149badf73bca Merged revisions 80382 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 59171 diff changeset	651 b'[&#56448;]')
149badf73bca Merged revisions 80382 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 59171 diff changeset	652 self.assertEqual("[\uDC80]".encode("utf-8", "surrogateescape"),
149badf73bca Merged revisions 80382 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 59171 diff changeset	653 b'[\x80]')
149badf73bca Merged revisions 80382 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 59171 diff changeset	654 self.assertEqual("[\uDC80]".encode("utf-8", "ignore"),
149badf73bca Merged revisions 80382 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 59171 diff changeset	655 b'[]')
149badf73bca Merged revisions 80382 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 59171 diff changeset	656 self.assertEqual("[\uDC80]".encode("utf-8", "replace"),
149badf73bca Merged revisions 80382 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 59171 diff changeset	657 b'[?]')
53681 52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 50975 diff changeset	658
53965 fbcdae3e0ca1 Rename the surrogates error handler to surrogatepass. Martin v. Löwis <martin@v.loewis.de> parents: 53859 diff changeset	659 def test_surrogatepass_handler(self):
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	660 self.assertEqual("abc\ud800def".encode("utf-8", "surrogatepass"),
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	661 b"abc\xed\xa0\x80def")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	662 self.assertEqual(b"abc\xed\xa0\x80def".decode("utf-8", "surrogatepass"),
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	663 "abc\ud800def")
72475 8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71512 diff changeset	664 self.assertEqual("\U00010fff\uD800".encode("utf-8", "surrogatepass"),
8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71512 diff changeset	665 b"\xf0\x90\xbf\xbf\xed\xa0\x80")
8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71512 diff changeset	666 self.assertEqual(b"\xf0\x90\xbf\xbf\xed\xa0\x80".decode("utf-8", "surrogatepass"),
8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71512 diff changeset	667 "\U00010fff\uD800")
53965 fbcdae3e0ca1 Rename the surrogates error handler to surrogatepass. Martin v. Löwis <martin@v.loewis.de> parents: 53859 diff changeset	668 self.assertTrue(codecs.lookup_error("surrogatepass"))
53681 52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 50975 diff changeset	669
73138 2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	670 @unittest.skipUnless(sys.platform == 'win32',
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	671 'cp65001 is a Windows-only codec')
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	672 class CP65001Test(ReadTest):
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	673 encoding = "cp65001"
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	674
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	675 def test_encode(self):
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	676 tests = [
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	677 ('abc', 'strict', b'abc'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	678 ('\xe9\u20ac', 'strict', b'\xc3\xa9\xe2\x82\xac'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	679 ('\U0010ffff', 'strict', b'\xf4\x8f\xbf\xbf'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	680 ]
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	681 if VISTA_OR_LATER:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	682 tests.extend((
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	683 ('\udc80', 'strict', None),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	684 ('\udc80', 'ignore', b''),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	685 ('\udc80', 'replace', b'?'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	686 ('\udc80', 'backslashreplace', b'\\udc80'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	687 ('\udc80', 'surrogatepass', b'\xed\xb2\x80'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	688 ))
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	689 else:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	690 tests.append(('\udc80', 'strict', b'\xed\xb2\x80'))
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	691 for text, errors, expected in tests:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	692 if expected is not None:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	693 try:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	694 encoded = text.encode('cp65001', errors)
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	695 except UnicodeEncodeError as err:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	696 self.fail('Unable to encode %a to cp65001 with '
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	697 'errors=%r: %s' % (text, errors, err))
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	698 self.assertEqual(encoded, expected,
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	699 '%a.encode("cp65001", %r)=%a != %a'
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	700 % (text, errors, encoded, expected))
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	701 else:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	702 self.assertRaises(UnicodeEncodeError,
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	703 text.encode, "cp65001", errors)
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	704
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	705 def test_decode(self):
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	706 tests = [
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	707 (b'abc', 'strict', 'abc'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	708 (b'\xc3\xa9\xe2\x82\xac', 'strict', '\xe9\u20ac'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	709 (b'\xf4\x8f\xbf\xbf', 'strict', '\U0010ffff'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	710 (b'\xef\xbf\xbd', 'strict', '\ufffd'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	711 (b'[\xc3\xa9]', 'strict', '[\xe9]'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	712 # invalid bytes
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	713 (b'[\xff]', 'strict', None),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	714 (b'[\xff]', 'ignore', '[]'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	715 (b'[\xff]', 'replace', '[\ufffd]'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	716 (b'[\xff]', 'surrogateescape', '[\udcff]'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	717 ]
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	718 if VISTA_OR_LATER:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	719 tests.extend((
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	720 (b'[\xed\xb2\x80]', 'strict', None),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	721 (b'[\xed\xb2\x80]', 'ignore', '[]'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	722 (b'[\xed\xb2\x80]', 'replace', '[\ufffd\ufffd\ufffd]'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	723 ))
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	724 else:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	725 tests.extend((
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	726 (b'[\xed\xb2\x80]', 'strict', '[\udc80]'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	727 ))
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	728 for raw, errors, expected in tests:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	729 if expected is not None:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	730 try:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	731 decoded = raw.decode('cp65001', errors)
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	732 except UnicodeDecodeError as err:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	733 self.fail('Unable to decode %a from cp65001 with '
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	734 'errors=%r: %s' % (raw, errors, err))
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	735 self.assertEqual(decoded, expected,
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	736 '%a.decode("cp65001", %r)=%a != %a'
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	737 % (raw, errors, decoded, expected))
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	738 else:
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	739 self.assertRaises(UnicodeDecodeError,
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	740 raw.decode, 'cp65001', errors)
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	741
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	742 @unittest.skipUnless(VISTA_OR_LATER, 'require Windows Vista or later')
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	743 def test_lone_surrogates(self):
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	744 self.assertRaises(UnicodeEncodeError, "\ud800".encode, "cp65001")
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	745 self.assertRaises(UnicodeDecodeError, b"\xed\xa0\x80".decode, "cp65001")
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	746 self.assertEqual("[\uDC80]".encode("cp65001", "backslashreplace"),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	747 b'[\\udc80]')
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	748 self.assertEqual("[\uDC80]".encode("cp65001", "xmlcharrefreplace"),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	749 b'[&#56448;]')
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	750 self.assertEqual("[\uDC80]".encode("cp65001", "surrogateescape"),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	751 b'[\x80]')
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	752 self.assertEqual("[\uDC80]".encode("cp65001", "ignore"),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	753 b'[]')
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	754 self.assertEqual("[\uDC80]".encode("cp65001", "replace"),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	755 b'[?]')
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	756
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	757 @unittest.skipUnless(VISTA_OR_LATER, 'require Windows Vista or later')
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	758 def test_surrogatepass_handler(self):
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	759 self.assertEqual("abc\ud800def".encode("cp65001", "surrogatepass"),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	760 b"abc\xed\xa0\x80def")
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	761 self.assertEqual(b"abc\xed\xa0\x80def".decode("cp65001", "surrogatepass"),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	762 "abc\ud800def")
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	763 self.assertEqual("\U00010fff\uD800".encode("cp65001", "surrogatepass"),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	764 b"\xf0\x90\xbf\xbf\xed\xa0\x80")
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	765 self.assertEqual(b"\xf0\x90\xbf\xbf\xed\xa0\x80".decode("cp65001", "surrogatepass"),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	766 "\U00010fff\uD800")
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	767 self.assertTrue(codecs.lookup_error("surrogatepass"))
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	768
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	769
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	770
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	771 class UTF7Test(ReadTest):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	772 encoding = "utf-7"
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	773
43933 9c4bf18e5d30 Merged revisions 59056-59076 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 43912 diff changeset	774 def test_partial(self):
9c4bf18e5d30 Merged revisions 59056-59076 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 43912 diff changeset	775 self.check_partial(
9c4bf18e5d30 Merged revisions 59056-59076 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 43912 diff changeset	776 "a+-b",
9c4bf18e5d30 Merged revisions 59056-59076 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 43912 diff changeset	777 [
9c4bf18e5d30 Merged revisions 59056-59076 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 43912 diff changeset	778 "a",
9c4bf18e5d30 Merged revisions 59056-59076 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 43912 diff changeset	779 "a",
9c4bf18e5d30 Merged revisions 59056-59076 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 43912 diff changeset	780 "a+",
9c4bf18e5d30 Merged revisions 59056-59076 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 43912 diff changeset	781 "a+-",
9c4bf18e5d30 Merged revisions 59056-59076 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 43912 diff changeset	782 "a+-b",
9c4bf18e5d30 Merged revisions 59056-59076 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 43912 diff changeset	783 ]
9c4bf18e5d30 Merged revisions 59056-59076 via svnmerge from Christian Heimes <christian@cheimes.de> parents: 43912 diff changeset	784 )
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	785
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	786 class UTF16ExTest(unittest.TestCase):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	787
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	788 def test_errors(self):
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	789 self.assertRaises(UnicodeDecodeError, codecs.utf_16_ex_decode, b"\xff", "strict", 0, True)
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	790
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	791 def test_bad_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	792 self.assertRaises(TypeError, codecs.utf_16_ex_decode)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	793
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	794 class ReadBufferTest(unittest.TestCase):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	795
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	796 def test_array(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	797 import array
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	798 self.assertEqual(
42911 cb28e14ee808 Changes in anticipation of stricter str vs. bytes enforcement. Guido van Rossum <guido@python.org> parents: 42626 diff changeset	799 codecs.readbuffer_encode(array.array("b", b"spam")),
42124 4590129e4c2b Change readbuffer_encode() and charbuffer_encode() to Walter Dörwald <walter@livinglogic.de> parents: 42118 diff changeset	800 (b"spam", 4)
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	801 )
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	802
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	803 def test_empty(self):
42124 4590129e4c2b Change readbuffer_encode() and charbuffer_encode() to Walter Dörwald <walter@livinglogic.de> parents: 42118 diff changeset	804 self.assertEqual(codecs.readbuffer_encode(""), (b"", 0))
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	805
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	806 def test_bad_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	807 self.assertRaises(TypeError, codecs.readbuffer_encode)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	808 self.assertRaises(TypeError, codecs.readbuffer_encode, 42)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	809
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	810 class UTF8SigTest(ReadTest):
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	811 encoding = "utf-8-sig"
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	812
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	813 def test_partial(self):
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	814 self.check_partial(
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	815 "\ufeff\x00\xff\u07ff\u0800\uffff",
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	816 [
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	817 "",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	818 "",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	819 "", # First BOM has been read and skipped
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	820 "",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	821 "",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	822 "\ufeff", # Second BOM has been read and emitted
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	823 "\ufeff\x00", # "\x00" read and emitted
42118 70a0831feda6 Patch by Ron Adam: Don't use u prefix in unicode error messages Walter Dörwald <walter@livinglogic.de> parents: 42064 diff changeset	824 "\ufeff\x00", # First byte of encoded "\xff" read
70a0831feda6 Patch by Ron Adam: Don't use u prefix in unicode error messages Walter Dörwald <walter@livinglogic.de> parents: 42064 diff changeset	825 "\ufeff\x00\xff", # Second byte of encoded "\xff" read
70a0831feda6 Patch by Ron Adam: Don't use u prefix in unicode error messages Walter Dörwald <walter@livinglogic.de> parents: 42064 diff changeset	826 "\ufeff\x00\xff", # First byte of encoded "\u07ff" read
70a0831feda6 Patch by Ron Adam: Don't use u prefix in unicode error messages Walter Dörwald <walter@livinglogic.de> parents: 42064 diff changeset	827 "\ufeff\x00\xff\u07ff", # Second byte of encoded "\u07ff" read
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	828 "\ufeff\x00\xff\u07ff",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	829 "\ufeff\x00\xff\u07ff",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	830 "\ufeff\x00\xff\u07ff\u0800",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	831 "\ufeff\x00\xff\u07ff\u0800",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	832 "\ufeff\x00\xff\u07ff\u0800",
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	833 "\ufeff\x00\xff\u07ff\u0800\uffff",
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	834 ]
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	835 )
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	836
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	837 def test_bug1601501(self):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	838 # SF bug #1601501: check that the codec works with a buffer
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	839 self.assertEqual(str(b"\xef\xbb\xbf", "utf-8-sig"), "")
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	840
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	841 def test_bom(self):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	842 d = codecs.getincrementaldecoder("utf-8-sig")()
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	843 s = "spam"
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	844 self.assertEqual(d.decode(s.encode("utf-8-sig")), s)
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	845
43912 4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	846 def test_stream_bom(self):
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	847 unistring = "ABC\u00A1\u2200XYZ"
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	848 bytestring = codecs.BOM_UTF8 + b"ABC\xC2\xA1\xE2\x88\x80XYZ"
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	849
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	850 reader = codecs.getreader("utf-8-sig")
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	851 for sizehint in [None] + list(range(1, 11)) + \
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	852 [64, 128, 256, 512, 1024]:
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	853 istream = reader(io.BytesIO(bytestring))
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	854 ostream = io.StringIO()
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	855 while 1:
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	856 if sizehint is not None:
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	857 data = istream.read(sizehint)
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	858 else:
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	859 data = istream.read()
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	860
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	861 if not data:
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	862 break
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	863 ostream.write(data)
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	864
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	865 got = ostream.getvalue()
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	866 self.assertEqual(got, unistring)
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	867
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	868 def test_stream_bare(self):
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	869 unistring = "ABC\u00A1\u2200XYZ"
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	870 bytestring = b"ABC\xC2\xA1\xE2\x88\x80XYZ"
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	871
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	872 reader = codecs.getreader("utf-8-sig")
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	873 for sizehint in [None] + list(range(1, 11)) + \
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	874 [64, 128, 256, 512, 1024]:
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	875 istream = reader(io.BytesIO(bytestring))
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	876 ostream = io.StringIO()
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	877 while 1:
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	878 if sizehint is not None:
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	879 data = istream.read(sizehint)
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	880 else:
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	881 data = istream.read()
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	882
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	883 if not data:
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	884 break
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	885 ostream.write(data)
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	886
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	887 got = ostream.getvalue()
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	888 self.assertEqual(got, unistring)
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	889
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	890 class EscapeDecodeTest(unittest.TestCase):
4747ef9640ae Merged revisions 59041-59055 via svnmerge from Guido van Rossum <guido@python.org> parents: 43777 diff changeset	891 def test_empty(self):
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	892 self.assertEqual(codecs.escape_decode(""), ("", 0))
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	893
27680 b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	894 class RecodingTest(unittest.TestCase):
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	895 def test_recoding(self):
41757 13a6c6bef609 Make test_codecs work. The CJK codecs now use bytes instead of str8 for Guido van Rossum <guido@python.org> parents: 41750 diff changeset	896 f = io.BytesIO()
27680 b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	897 f2 = codecs.EncodedFile(f, "unicode_internal", "utf-8")
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	898 f2.write("a")
27680 b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	899 f2.close()
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	900 # Python used to crash on this at exit because of a refcount
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	901 # bug in _codecsmodule.c
20013 e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	902
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	903 # From RFC 3492
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	904 punycode_testcases = [
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	905 # A Arabic (Egyptian):
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	906 ("\u0644\u064A\u0647\u0645\u0627\u0628\u062A\u0643\u0644"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	907 "\u0645\u0648\u0634\u0639\u0631\u0628\u064A\u061F",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	908 b"egbpdaj6bu4bxfgehfvwxn"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	909 # B Chinese (simplified):
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	910 ("\u4ED6\u4EEC\u4E3A\u4EC0\u4E48\u4E0D\u8BF4\u4E2D\u6587",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	911 b"ihqwcrb4cv8a8dqg056pqjye"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	912 # C Chinese (traditional):
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	913 ("\u4ED6\u5011\u7232\u4EC0\u9EBD\u4E0D\u8AAA\u4E2D\u6587",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	914 b"ihqwctvzc91f659drss3x8bo0yb"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	915 # D Czech: Pro<ccaron>prost<ecaron>nemluv<iacute><ccaron>esky
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	916 ("\u0050\u0072\u006F\u010D\u0070\u0072\u006F\u0073\u0074"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	917 "\u011B\u006E\u0065\u006D\u006C\u0075\u0076\u00ED\u010D"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	918 "\u0065\u0073\u006B\u0079",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	919 b"Proprostnemluvesky-uyb24dma41a"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	920 # E Hebrew:
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	921 ("\u05DC\u05DE\u05D4\u05D4\u05DD\u05E4\u05E9\u05D5\u05D8"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	922 "\u05DC\u05D0\u05DE\u05D3\u05D1\u05E8\u05D9\u05DD\u05E2"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	923 "\u05D1\u05E8\u05D9\u05EA",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	924 b"4dbcagdahymbxekheh6e0a7fei0b"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	925 # F Hindi (Devanagari):
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	926 ("\u092F\u0939\u0932\u094B\u0917\u0939\u093F\u0928\u094D"
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	927 "\u0926\u0940\u0915\u094D\u092F\u094B\u0902\u0928\u0939"
98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	928 "\u0940\u0902\u092C\u094B\u0932\u0938\u0915\u0924\u0947"
98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	929 "\u0939\u0948\u0902",
98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	930 b"i1baa7eci9glrd9b2ae1bj0hfcgg6iyaf8o0a1dig0cd"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	931
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	932 #(G) Japanese (kanji and hiragana):
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	933 ("\u306A\u305C\u307F\u3093\u306A\u65E5\u672C\u8A9E\u3092"
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	934 "\u8A71\u3057\u3066\u304F\u308C\u306A\u3044\u306E\u304B",
98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	935 b"n8jok5ay5dzabd5bym9f0cm5685rrjetr6pdxa"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	936
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	937 # (H) Korean (Hangul syllables):
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	938 ("\uC138\uACC4\uC758\uBAA8\uB4E0\uC0AC\uB78C\uB4E4\uC774"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	939 "\uD55C\uAD6D\uC5B4\uB97C\uC774\uD574\uD55C\uB2E4\uBA74"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	940 "\uC5BC\uB9C8\uB098\uC88B\uC744\uAE4C",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	941 b"989aomsvi5e83db1d2a355cv1e0vak1dwrv93d5xbh15a0dt30a5j"
98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	942 b"psd879ccm6fea98c"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	943
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	944 # (I) Russian (Cyrillic):
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	945 ("\u043F\u043E\u0447\u0435\u043C\u0443\u0436\u0435\u043E"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	946 "\u043D\u0438\u043D\u0435\u0433\u043E\u0432\u043E\u0440"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	947 "\u044F\u0442\u043F\u043E\u0440\u0443\u0441\u0441\u043A"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	948 "\u0438",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	949 b"b1abfaaepdrnnbgefbaDotcwatmq2g4l"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	950
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	951 # (J) Spanish: Porqu<eacute>nopuedensimplementehablarenEspa<ntilde>ol
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	952 ("\u0050\u006F\u0072\u0071\u0075\u00E9\u006E\u006F\u0070"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	953 "\u0075\u0065\u0064\u0065\u006E\u0073\u0069\u006D\u0070"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	954 "\u006C\u0065\u006D\u0065\u006E\u0074\u0065\u0068\u0061"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	955 "\u0062\u006C\u0061\u0072\u0065\u006E\u0045\u0073\u0070"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	956 "\u0061\u00F1\u006F\u006C",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	957 b"PorqunopuedensimplementehablarenEspaol-fmd56a"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	958
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	959 # (K) Vietnamese:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	960 # T<adotbelow>isaoh<odotbelow>kh<ocirc>ngth<ecirchookabove>ch\
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	961 # <ihookabove>n<oacute>iti<ecircacute>ngVi<ecircdotbelow>t
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	962 ("\u0054\u1EA1\u0069\u0073\u0061\u006F\u0068\u1ECD\u006B"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	963 "\u0068\u00F4\u006E\u0067\u0074\u0068\u1EC3\u0063\u0068"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	964 "\u1EC9\u006E\u00F3\u0069\u0074\u0069\u1EBF\u006E\u0067"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	965 "\u0056\u0069\u1EC7\u0074",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	966 b"TisaohkhngthchnitingVit-kjcr8268qyxafd2f1b9g"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	967
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	968 #(L) 3<nen>B<gumi><kinpachi><sensei>
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	969 ("\u0033\u5E74\u0042\u7D44\u91D1\u516B\u5148\u751F",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	970 b"3B-ww4c5e180e575a65lsy2b"),
28774 c9bee11eaa8b Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 28678 diff changeset	971
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	972 # (M) <amuro><namie>-with-SUPER-MONKEYS
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	973 ("\u5B89\u5BA4\u5948\u7F8E\u6075\u002D\u0077\u0069\u0074"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	974 "\u0068\u002D\u0053\u0055\u0050\u0045\u0052\u002D\u004D"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	975 "\u004F\u004E\u004B\u0045\u0059\u0053",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	976 b"-with-SUPER-MONKEYS-pc58ag80a8qai00g7n9n"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	977
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	978 # (N) Hello-Another-Way-<sorezore><no><basho>
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	979 ("\u0048\u0065\u006C\u006C\u006F\u002D\u0041\u006E\u006F"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	980 "\u0074\u0068\u0065\u0072\u002D\u0057\u0061\u0079\u002D"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	981 "\u305D\u308C\u305E\u308C\u306E\u5834\u6240",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	982 b"Hello-Another-Way--fc4qua05auwb3674vfr0b"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	983
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	984 # (O) <hitotsu><yane><no><shita>2
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	985 ("\u3072\u3068\u3064\u5C4B\u6839\u306E\u4E0B\u0032",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	986 b"2-u9tlzr9756bt3uc0v"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	987
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	988 # (P) Maji<de>Koi<suru>5<byou><mae>
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	989 ("\u004D\u0061\u006A\u0069\u3067\u004B\u006F\u0069\u3059"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	990 "\u308B\u0035\u79D2\u524D",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	991 b"MajiKoi5-783gue6qz075azm5e"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	992
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	993 # (Q) <pafii>de<runba>
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	994 ("\u30D1\u30D5\u30A3\u30FC\u0064\u0065\u30EB\u30F3\u30D0",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	995 b"de-jg4avhby1noc0d"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	996
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	997 # (R) <sono><supiido><de>
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	998 ("\u305D\u306E\u30B9\u30D4\u30FC\u30C9\u3067",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	999 b"d9juau41awczczp"),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1000
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1001 # (S) -> $1.00 <-
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1002 ("\u002D\u003E\u0020\u0024\u0031\u002E\u0030\u0030\u0020"
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1003 "\u003C\u002D",
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1004 b"-> $1.00 <--")
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1005 ]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1006
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1007 for i in punycode_testcases:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1008 if len(i)!=2:
40775 4cdf21a0dbbc Fix most trivially-findable print statements. Guido van Rossum <guido@python.org> parents: 40587 diff changeset	1009 print(repr(i))
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1010
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1011 class PunycodeTest(unittest.TestCase):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1012 def test_encode(self):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1013 for uni, puny in punycode_testcases:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1014 # Need to convert both strings to lower case, since
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1015 # some of the extended encodings use upper case, but our
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1016 # code produces only lower case. Converting just puny to
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1017 # lower is also insufficient, since some of the input characters
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1018 # are upper case.
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1019 self.assertEqual(
41679 98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1020 str(uni.encode("punycode"), "ascii").lower(),
98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1021 str(puny, "ascii").lower()
98709189d9e1 Fix punycode codec and tests. Walter Dörwald <walter@livinglogic.de> parents: 41676 diff changeset	1022 )
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1023
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1024 def test_decode(self):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1025 for uni, puny in punycode_testcases:
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1026 self.assertEqual(uni, puny.decode("punycode"))
42985 f4851dc0df04 Fix the one failing test (can't decode twice). Guido van Rossum <guido@python.org> parents: 42911 diff changeset	1027 puny = puny.decode("ascii").encode("ascii")
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1028 self.assertEqual(uni, puny.decode("punycode"))
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1029
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	1030 class UnicodeInternalTest(unittest.TestCase):
72534 85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1031 @unittest.skipUnless(SIZEOF_WCHAR_T == 4, 'specific to 32-bit wchar_t')
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	1032 def test_bug1251300(self):
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	1033 # Decoding with unicode_internal used to not correctly handle "code
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	1034 # points" above 0x10ffff on UCS-4 builds.
72534 85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1035 ok = [
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1036 (b"\x00\x10\xff\xff", "\U0010ffff"),
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1037 (b"\x00\x00\x01\x01", "\U00000101"),
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1038 (b"", ""),
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1039 ]
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1040 not_ok = [
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1041 b"\x7f\xff\xff\xff",
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1042 b"\x80\x00\x00\x00",
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1043 b"\x81\x00\x00\x00",
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1044 b"\x00",
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1045 b"\x00\x00\x00\x00\x00",
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1046 ]
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1047 for internal, uni in ok:
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1048 if sys.byteorder == "little":
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1049 internal = bytes(reversed(internal))
73588 4f534cd40f54 #13406: silence deprecation warnings in test_codecs. Ezio Melotti <ezio.melotti@gmail.com> parents: 73579 diff changeset	1050 with support.check_warnings():
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1051 self.assertEqual(uni, internal.decode("unicode_internal"))
72534 85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1052 for internal in not_ok:
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1053 if sys.byteorder == "little":
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1054 internal = bytes(reversed(internal))
73589 028b61c4f926 #13406: correct the error message in check_warnings too. Ezio Melotti <ezio.melotti@gmail.com> parents: 73588 diff changeset	1055 with support.check_warnings(('unicode_internal codec has been '
73588 4f534cd40f54 #13406: silence deprecation warnings in test_codecs. Ezio Melotti <ezio.melotti@gmail.com> parents: 73579 diff changeset	1056 'deprecated', DeprecationWarning)):
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	1057 self.assertRaises(UnicodeDecodeError, internal.decode,
73588 4f534cd40f54 #13406: silence deprecation warnings in test_codecs. Ezio Melotti <ezio.melotti@gmail.com> parents: 73579 diff changeset	1058 "unicode_internal")
73906 bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	1059 if sys.byteorder == "little":
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	1060 invalid = b"\x00\x00\x11\x00"
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	1061 else:
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	1062 invalid = b"\x00\x11\x00\x00"
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	1063 with support.check_warnings():
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	1064 self.assertRaises(UnicodeDecodeError,
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	1065 invalid.decode, "unicode_internal")
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	1066 with support.check_warnings():
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	1067 self.assertEqual(invalid.decode("unicode_internal", "replace"),
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	1068 '\ufffd')
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	1069
72534 85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1070 @unittest.skipUnless(SIZEOF_WCHAR_T == 4, 'specific to 32-bit wchar_t')
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	1071 def test_decode_error_attributes(self):
72534 85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1072 try:
73589 028b61c4f926 #13406: correct the error message in check_warnings too. Ezio Melotti <ezio.melotti@gmail.com> parents: 73588 diff changeset	1073 with support.check_warnings(('unicode_internal codec has been '
73588 4f534cd40f54 #13406: silence deprecation warnings in test_codecs. Ezio Melotti <ezio.melotti@gmail.com> parents: 73579 diff changeset	1074 'deprecated', DeprecationWarning)):
41971 a3261cde3a37 Fix tests for unicode-internal codec. Walter Dörwald <walter@livinglogic.de> parents: 41757 diff changeset	1075 b"\x00\x00\x00\x00\x00\x11\x11\x00".decode("unicode_internal")
72534 85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1076 except UnicodeDecodeError as ex:
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1077 self.assertEqual("unicode_internal", ex.encoding)
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1078 self.assertEqual(b"\x00\x00\x00\x00\x00\x11\x11\x00", ex.object)
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1079 self.assertEqual(4, ex.start)
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1080 self.assertEqual(8, ex.end)
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1081 else:
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1082 self.fail()
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	1083
72534 85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1084 @unittest.skipUnless(SIZEOF_WCHAR_T == 4, 'specific to 32-bit wchar_t')
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	1085 def test_decode_callback(self):
72534 85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1086 codecs.register_error("UnicodeInternalTest", codecs.ignore_errors)
85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1087 decoder = codecs.getdecoder("unicode_internal")
73589 028b61c4f926 #13406: correct the error message in check_warnings too. Ezio Melotti <ezio.melotti@gmail.com> parents: 73588 diff changeset	1088 with support.check_warnings(('unicode_internal codec has been '
73588 4f534cd40f54 #13406: silence deprecation warnings in test_codecs. Ezio Melotti <ezio.melotti@gmail.com> parents: 73579 diff changeset	1089 'deprecated', DeprecationWarning)):
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43577 diff changeset	1090 ab = "ab".encode("unicode_internal").decode()
43577 ace45b361e87 Patch# 1258 by Christian Heimes: kill basestring. Guido van Rossum <guido@python.org> parents: 42985 diff changeset	1091 ignored = decoder(bytes("%s\x22\x22\x22\x22%s" % (ab[:4], ab[4:]),
ace45b361e87 Patch# 1258 by Christian Heimes: kill basestring. Guido van Rossum <guido@python.org> parents: 42985 diff changeset	1092 "ascii"),
ace45b361e87 Patch# 1258 by Christian Heimes: kill basestring. Guido van Rossum <guido@python.org> parents: 42985 diff changeset	1093 "UnicodeInternalTest")
72534 85d11cf67aa8 Fix test_codecs for Windows: check size of wchar_t, not sys.maxunicode Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1094 self.assertEqual(("ab", 12), ignored)
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	1095
53859 63b23bd3904d Merged revisions 72404-72406 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 53767 diff changeset	1096 def test_encode_length(self):
73600 fe2be7d35660 #13406: fix more deprecation warnings and move the deprecation of unicode-internal earlier in the code. Ezio Melotti <ezio.melotti@gmail.com> parents: 73589 diff changeset	1097 with support.check_warnings(('unicode_internal codec has been '
fe2be7d35660 #13406: fix more deprecation warnings and move the deprecation of unicode-internal earlier in the code. Ezio Melotti <ezio.melotti@gmail.com> parents: 73589 diff changeset	1098 'deprecated', DeprecationWarning)):
73579 ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	1099 # Issue 3739
ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	1100 encoder = codecs.getencoder("unicode_internal")
ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	1101 self.assertEqual(encoder("a")[1], 1)
ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	1102 self.assertEqual(encoder("\xe9\u0142")[1], 2)
53859 63b23bd3904d Merged revisions 72404-72406 via svnmerge from Walter Dörwald <walter@livinglogic.de> parents: 53767 diff changeset	1103
73579 ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	1104 self.assertEqual(codecs.escape_encode(br'\x00')[1], 4)
62215 f33e464a9a60 Merged revisions 79780 via svnmerge from Philip Jenvey <pjenvey@underboss.org> parents: 61863 diff changeset	1105
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1106 # From http://www.gnu.org/software/libidn/draft-josefsson-idn-test-vectors.html
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1107 nameprep_tests = [
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1108 # 3.1 Map to nothing.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1109 (b'foo\xc2\xad\xcd\x8f\xe1\xa0\x86\xe1\xa0\x8bbar'
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1110 b'\xe2\x80\x8b\xe2\x81\xa0baz\xef\xb8\x80\xef\xb8\x88\xef'
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1111 b'\xb8\x8f\xef\xbb\xbf',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1112 b'foobarbaz'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1113 # 3.2 Case folding ASCII U+0043 U+0041 U+0046 U+0045.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1114 (b'CAFE',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1115 b'cafe'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1116 # 3.3 Case folding 8bit U+00DF (german sharp s).
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1117 # The original test case is bogus; it says \xc3\xdf
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1118 (b'\xc3\x9f',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1119 b'ss'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1120 # 3.4 Case folding U+0130 (turkish capital I with dot).
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1121 (b'\xc4\xb0',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1122 b'i\xcc\x87'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1123 # 3.5 Case folding multibyte U+0143 U+037A.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1124 (b'\xc5\x83\xcd\xba',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1125 b'\xc5\x84 \xce\xb9'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1126 # 3.6 Case folding U+2121 U+33C6 U+1D7BB.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1127 # XXX: skip this as it fails in UCS-2 mode
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1128 #('\xe2\x84\xa1\xe3\x8f\x86\xf0\x9d\x9e\xbb',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1129 # 'telc\xe2\x88\x95kg\xcf\x83'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1130 (None, None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1131 # 3.7 Normalization of U+006a U+030c U+00A0 U+00AA.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1132 (b'j\xcc\x8c\xc2\xa0\xc2\xaa',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1133 b'\xc7\xb0 a'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1134 # 3.8 Case folding U+1FB7 and normalization.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1135 (b'\xe1\xbe\xb7',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1136 b'\xe1\xbe\xb6\xce\xb9'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1137 # 3.9 Self-reverting case folding U+01F0 and normalization.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1138 # The original test case is bogus, it says `\xc7\xf0'
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1139 (b'\xc7\xb0',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1140 b'\xc7\xb0'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1141 # 3.10 Self-reverting case folding U+0390 and normalization.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1142 (b'\xce\x90',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1143 b'\xce\x90'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1144 # 3.11 Self-reverting case folding U+03B0 and normalization.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1145 (b'\xce\xb0',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1146 b'\xce\xb0'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1147 # 3.12 Self-reverting case folding U+1E96 and normalization.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1148 (b'\xe1\xba\x96',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1149 b'\xe1\xba\x96'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1150 # 3.13 Self-reverting case folding U+1F56 and normalization.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1151 (b'\xe1\xbd\x96',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1152 b'\xe1\xbd\x96'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1153 # 3.14 ASCII space character U+0020.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1154 (b' ',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1155 b' '),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1156 # 3.15 Non-ASCII 8bit space character U+00A0.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1157 (b'\xc2\xa0',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1158 b' '),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1159 # 3.16 Non-ASCII multibyte space character U+1680.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1160 (b'\xe1\x9a\x80',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1161 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1162 # 3.17 Non-ASCII multibyte space character U+2000.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1163 (b'\xe2\x80\x80',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1164 b' '),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1165 # 3.18 Zero Width Space U+200b.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1166 (b'\xe2\x80\x8b',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1167 b''),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1168 # 3.19 Non-ASCII multibyte space character U+3000.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1169 (b'\xe3\x80\x80',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1170 b' '),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1171 # 3.20 ASCII control characters U+0010 U+007F.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1172 (b'\x10\x7f',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1173 b'\x10\x7f'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1174 # 3.21 Non-ASCII 8bit control character U+0085.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1175 (b'\xc2\x85',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1176 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1177 # 3.22 Non-ASCII multibyte control character U+180E.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1178 (b'\xe1\xa0\x8e',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1179 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1180 # 3.23 Zero Width No-Break Space U+FEFF.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1181 (b'\xef\xbb\xbf',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1182 b''),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1183 # 3.24 Non-ASCII control character U+1D175.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1184 (b'\xf0\x9d\x85\xb5',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1185 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1186 # 3.25 Plane 0 private use character U+F123.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1187 (b'\xef\x84\xa3',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1188 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1189 # 3.26 Plane 15 private use character U+F1234.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1190 (b'\xf3\xb1\x88\xb4',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1191 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1192 # 3.27 Plane 16 private use character U+10F234.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1193 (b'\xf4\x8f\x88\xb4',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1194 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1195 # 3.28 Non-character code point U+8FFFE.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1196 (b'\xf2\x8f\xbf\xbe',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1197 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1198 # 3.29 Non-character code point U+10FFFF.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1199 (b'\xf4\x8f\xbf\xbf',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1200 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1201 # 3.30 Surrogate code U+DF42.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1202 (b'\xed\xbd\x82',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1203 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1204 # 3.31 Non-plain text character U+FFFD.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1205 (b'\xef\xbf\xbd',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1206 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1207 # 3.32 Ideographic description character U+2FF5.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1208 (b'\xe2\xbf\xb5',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1209 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1210 # 3.33 Display property character U+0341.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1211 (b'\xcd\x81',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1212 b'\xcc\x81'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1213 # 3.34 Left-to-right mark U+200E.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1214 (b'\xe2\x80\x8e',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1215 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1216 # 3.35 Deprecated U+202A.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1217 (b'\xe2\x80\xaa',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1218 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1219 # 3.36 Language tagging character U+E0001.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1220 (b'\xf3\xa0\x80\x81',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1221 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1222 # 3.37 Language tagging character U+E0042.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1223 (b'\xf3\xa0\x81\x82',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1224 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1225 # 3.38 Bidi: RandALCat character U+05BE and LCat characters.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1226 (b'foo\xd6\xbebar',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1227 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1228 # 3.39 Bidi: RandALCat character U+FD50 and LCat characters.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1229 (b'foo\xef\xb5\x90bar',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1230 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1231 # 3.40 Bidi: RandALCat character U+FB38 and LCat characters.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1232 (b'foo\xef\xb9\xb6bar',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1233 b'foo \xd9\x8ebar'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1234 # 3.41 Bidi: RandALCat without trailing RandALCat U+0627 U+0031.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1235 (b'\xd8\xa71',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1236 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1237 # 3.42 Bidi: RandALCat character U+0627 U+0031 U+0628.
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1238 (b'\xd8\xa71\xd8\xa8',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1239 b'\xd8\xa71\xd8\xa8'),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1240 # 3.43 Unassigned code point U+E0002.
28678 31cb8e0260db Skip nameprep test 3.43, as we do allow unassigned characters. The test Martin v. Löwis <martin@v.loewis.de> parents: 28671 diff changeset	1241 # Skip this test as we allow unassigned
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1242 #(b'\xf3\xa0\x80\x82',
28678 31cb8e0260db Skip nameprep test 3.43, as we do allow unassigned characters. The test Martin v. Löwis <martin@v.loewis.de> parents: 28671 diff changeset	1243 # None),
31cb8e0260db Skip nameprep test 3.43, as we do allow unassigned characters. The test Martin v. Löwis <martin@v.loewis.de> parents: 28671 diff changeset	1244 (None, None),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1245 # 3.44 Larger test (shrinking).
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1246 # Original test case reads \xc3\xdf
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1247 (b'X\xc2\xad\xc3\x9f\xc4\xb0\xe2\x84\xa1j\xcc\x8c\xc2\xa0\xc2'
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1248 b'\xaa\xce\xb0\xe2\x80\x80',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1249 b'xssi\xcc\x87tel\xc7\xb0 a\xce\xb0 '),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1250 # 3.45 Larger test (expanding).
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1251 # Original test case reads \xc3\x9f
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1252 (b'X\xc3\x9f\xe3\x8c\x96\xc4\xb0\xe2\x84\xa1\xe2\x92\x9f\xe3\x8c'
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1253 b'\x80',
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1254 b'xss\xe3\x82\xad\xe3\x83\xad\xe3\x83\xa1\xe3\x83\xbc\xe3'
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1255 b'\x83\x88\xe3\x83\xabi\xcc\x87tel\x28d\x29\xe3\x82'
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1256 b'\xa2\xe3\x83\x91\xe3\x83\xbc\xe3\x83\x88')
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1257 ]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1258
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1259
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1260 class NameprepTest(unittest.TestCase):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1261 def test_nameprep(self):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1262 from encodings.idna import nameprep
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1263 for pos, (orig, prepped) in enumerate(nameprep_tests):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1264 if orig is None:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1265 # Skipped
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1266 continue
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1267 # The Unicode strings are given in UTF-8
53965 fbcdae3e0ca1 Rename the surrogates error handler to surrogatepass. Martin v. Löwis <martin@v.loewis.de> parents: 53859 diff changeset	1268 orig = str(orig, "utf-8", "surrogatepass")
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1269 if prepped is None:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1270 # Input contains prohibited characters
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1271 self.assertRaises(UnicodeError, nameprep, orig)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1272 else:
53965 fbcdae3e0ca1 Rename the surrogates error handler to surrogatepass. Martin v. Löwis <martin@v.loewis.de> parents: 53859 diff changeset	1273 prepped = str(prepped, "utf-8", "surrogatepass")
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1274 try:
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1275 self.assertEqual(nameprep(orig), prepped)
40587 6b18a095c2e7 SF patch 1631942 by Collin Winter: Guido van Rossum <guido@python.org> parents: 40450 diff changeset	1276 except Exception as e:
47201 9c930a32cebb #2621 rename test.test_support to test.support Benjamin Peterson <benjamin@python.org> parents: 43933 diff changeset	1277 raise support.TestFailed("Test 3.%d: %s" % (pos+1, str(e)))
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1278
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1279 class IDNACodecTest(unittest.TestCase):
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1280 def test_builtin_decode(self):
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1281 self.assertEqual(str(b"python.org", "idna"), "python.org")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1282 self.assertEqual(str(b"python.org.", "idna"), "python.org.")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1283 self.assertEqual(str(b"xn--pythn-mua.org", "idna"), "pyth\xf6n.org")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1284 self.assertEqual(str(b"xn--pythn-mua.org.", "idna"), "pyth\xf6n.org.")
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1285
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1286 def test_builtin_encode(self):
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1287 self.assertEqual("python.org".encode("idna"), b"python.org")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1288 self.assertEqual("python.org.".encode("idna"), b"python.org.")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1289 self.assertEqual("pyth\xf6n.org".encode("idna"), b"xn--pythn-mua.org")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1290 self.assertEqual("pyth\xf6n.org.".encode("idna"), b"xn--pythn-mua.org.")
31795 bf37429f6d9f Add test case for unicode(somestring, "idna"). Martin v. Löwis <martin@v.loewis.de> parents: 28917 diff changeset	1291
35435 9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	1292 def test_stream(self):
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	1293 r = codecs.getreader("idna")(io.BytesIO(b"abc"))
35435 9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	1294 r.read(3)
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1295 self.assertEqual(r.read(), "")
35435 9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	1296
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1297 def test_incremental_decode(self):
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1298 self.assertEqual(
42911 cb28e14ee808 Changes in anticipation of stricter str vs. bytes enforcement. Guido van Rossum <guido@python.org> parents: 42626 diff changeset	1299 "".join(codecs.iterdecode((bytes([c]) for c in b"python.org"), "idna")),
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1300 "python.org"
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1301 )
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1302 self.assertEqual(
42911 cb28e14ee808 Changes in anticipation of stricter str vs. bytes enforcement. Guido van Rossum <guido@python.org> parents: 42626 diff changeset	1303 "".join(codecs.iterdecode((bytes([c]) for c in b"python.org."), "idna")),
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1304 "python.org."
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1305 )
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1306 self.assertEqual(
42911 cb28e14ee808 Changes in anticipation of stricter str vs. bytes enforcement. Guido van Rossum <guido@python.org> parents: 42626 diff changeset	1307 "".join(codecs.iterdecode((bytes([c]) for c in b"xn--pythn-mua.org."), "idna")),
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1308 "pyth\xf6n.org."
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1309 )
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1310 self.assertEqual(
42911 cb28e14ee808 Changes in anticipation of stricter str vs. bytes enforcement. Guido van Rossum <guido@python.org> parents: 42626 diff changeset	1311 "".join(codecs.iterdecode((bytes([c]) for c in b"xn--pythn-mua.org."), "idna")),
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1312 "pyth\xf6n.org."
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1313 )
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1314
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1315 decoder = codecs.getincrementaldecoder("idna")()
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1316 self.assertEqual(decoder.decode(b"xn--xam", ), "")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1317 self.assertEqual(decoder.decode(b"ple-9ta.o", ), "\xe4xample.")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1318 self.assertEqual(decoder.decode(b"rg"), "")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1319 self.assertEqual(decoder.decode(b"", True), "org")
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1320
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1321 decoder.reset()
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1322 self.assertEqual(decoder.decode(b"xn--xam", ), "")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1323 self.assertEqual(decoder.decode(b"ple-9ta.o", ), "\xe4xample.")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1324 self.assertEqual(decoder.decode(b"rg."), "org.")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1325 self.assertEqual(decoder.decode(b"", True), "")
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1326
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1327 def test_incremental_encode(self):
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1328 self.assertEqual(
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1329 b"".join(codecs.iterencode("python.org", "idna")),
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1330 b"python.org"
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1331 )
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1332 self.assertEqual(
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1333 b"".join(codecs.iterencode("python.org.", "idna")),
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1334 b"python.org."
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1335 )
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1336 self.assertEqual(
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1337 b"".join(codecs.iterencode("pyth\xf6n.org.", "idna")),
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1338 b"xn--pythn-mua.org."
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1339 )
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1340 self.assertEqual(
41693 d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1341 b"".join(codecs.iterencode("pyth\xf6n.org.", "idna")),
d91cc8d49294 Enhance the punycode decoder so that it can decode Walter Dörwald <walter@livinglogic.de> parents: 41681 diff changeset	1342 b"xn--pythn-mua.org."
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1343 )
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1344
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1345 encoder = codecs.getincrementalencoder("idna")()
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1346 self.assertEqual(encoder.encode("\xe4x"), b"")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1347 self.assertEqual(encoder.encode("ample.org"), b"xn--xample-9ta.")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1348 self.assertEqual(encoder.encode("", True), b"org")
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1349
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1350 encoder.reset()
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1351 self.assertEqual(encoder.encode("\xe4x"), b"")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1352 self.assertEqual(encoder.encode("ample.org."), b"xn--xample-9ta.org.")
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1353 self.assertEqual(encoder.encode("", True), b"")
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1354
32572 8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1355 class CodecsModuleTest(unittest.TestCase):
8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1356
8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1357 def test_decode(self):
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1358 self.assertEqual(codecs.decode(b'\xe4\xf6\xfc', 'latin-1'),
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1359 '\xe4\xf6\xfc')
33855 1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1360 self.assertRaises(TypeError, codecs.decode)
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1361 self.assertEqual(codecs.decode(b'abc'), 'abc')
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1362 self.assertRaises(UnicodeDecodeError, codecs.decode, b'\xff', 'ascii')
33855 1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1363
32572 8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1364 def test_encode(self):
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1365 self.assertEqual(codecs.encode('\xe4\xf6\xfc', 'latin-1'),
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1366 b'\xe4\xf6\xfc')
33855 1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1367 self.assertRaises(TypeError, codecs.encode)
35889 e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1368 self.assertRaises(LookupError, codecs.encode, "foo", "__spam__")
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1369 self.assertEqual(codecs.encode('abc'), b'abc')
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1370 self.assertRaises(UnicodeEncodeError, codecs.encode, '\xffff', 'ascii')
33855 1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1371
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1372 def test_register(self):
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1373 self.assertRaises(TypeError, codecs.register)
35889 e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1374 self.assertRaises(TypeError, codecs.register, 42)
33855 1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1375
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1376 def test_lookup(self):
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1377 self.assertRaises(TypeError, codecs.lookup)
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1378 self.assertRaises(LookupError, codecs.lookup, "__spam__")
35889 e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1379 self.assertRaises(LookupError, codecs.lookup, " ")
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1380
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1381 def test_getencoder(self):
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1382 self.assertRaises(TypeError, codecs.getencoder)
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1383 self.assertRaises(LookupError, codecs.getencoder, "__spam__")
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1384
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1385 def test_getdecoder(self):
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1386 self.assertRaises(TypeError, codecs.getdecoder)
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1387 self.assertRaises(LookupError, codecs.getdecoder, "__spam__")
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1388
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1389 def test_getreader(self):
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1390 self.assertRaises(TypeError, codecs.getreader)
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1391 self.assertRaises(LookupError, codecs.getreader, "__spam__")
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1392
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1393 def test_getwriter(self):
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1394 self.assertRaises(TypeError, codecs.getwriter)
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1395 self.assertRaises(LookupError, codecs.getwriter, "__spam__")
32572 8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1396
71489 92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70293 diff changeset	1397 def test_lookup_issue1813(self):
92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70293 diff changeset	1398 # Issue #1813: under Turkish locales, lookup of some codecs failed
92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70293 diff changeset	1399 # because 'I' is lowercased as "ı" (dotless i)
71511 7063aa2fa127 Backport 0398f07d4827 (fix for weird buildbot failures) Antoine Pitrou <solipsis@pitrou.net> parents: 71489 diff changeset	1400 oldlocale = locale.setlocale(locale.LC_CTYPE)
71489 92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70293 diff changeset	1401 self.addCleanup(locale.setlocale, locale.LC_CTYPE, oldlocale)
92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70293 diff changeset	1402 try:
92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70293 diff changeset	1403 locale.setlocale(locale.LC_CTYPE, 'tr_TR')
92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70293 diff changeset	1404 except locale.Error:
92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70293 diff changeset	1405 # Unsupported locale on this system
92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70293 diff changeset	1406 self.skipTest('test needs Turkish locale')
92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70293 diff changeset	1407 c = codecs.lookup('ASCII')
92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70293 diff changeset	1408 self.assertEqual(c.name, 'ascii')
92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70293 diff changeset	1409
33782 a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1410 class StreamReaderTest(unittest.TestCase):
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1411
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1412 def setUp(self):
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1413 self.reader = codecs.getreader('utf-8')
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	1414 self.stream = io.BytesIO(b'\xed\x95\x9c\n\xea\xb8\x80')
33782 a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1415
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1416 def test_readlines(self):
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1417 f = self.reader(self.stream)
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1418 self.assertEqual(f.readlines(), ['\ud55c\n', '\uae00'])
33782 a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1419
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1420 class EncodedFileTest(unittest.TestCase):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1421
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1422 def test_basic(self):
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	1423 f = io.BytesIO(b'\xed\x95\x9c\n\xea\xb8\x80')
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1424 ef = codecs.EncodedFile(f, 'utf-16-le', 'utf-8')
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1425 self.assertEqual(ef.read(), b'\\\xd5\n\x00\x00\xae')
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1426
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	1427 f = io.BytesIO()
68142 be6c38d1817b Normalize the encoding names for Latin-1 and UTF-8 to Marc-André Lemburg <mal@egenix.com> parents: 66837 diff changeset	1428 ef = codecs.EncodedFile(f, 'utf-8', 'latin-1')
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1429 ef.write(b'\xc3\xbc')
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1430 self.assertEqual(f.getvalue(), b'\xfc')
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1431
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1432 all_unicode_encodings = [
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1433 "ascii",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1434 "big5",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1435 "big5hkscs",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1436 "charmap",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1437 "cp037",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1438 "cp1006",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1439 "cp1026",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1440 "cp1140",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1441 "cp1250",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1442 "cp1251",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1443 "cp1252",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1444 "cp1253",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1445 "cp1254",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1446 "cp1255",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1447 "cp1256",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1448 "cp1257",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1449 "cp1258",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1450 "cp424",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1451 "cp437",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1452 "cp500",
62572 1d1797af4622 Merged revisions 81499,81506 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 62367 diff changeset	1453 "cp720",
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1454 "cp737",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1455 "cp775",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1456 "cp850",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1457 "cp852",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1458 "cp855",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1459 "cp856",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1460 "cp857",
62572 1d1797af4622 Merged revisions 81499,81506 via svnmerge from Benjamin Peterson <benjamin@python.org> parents: 62367 diff changeset	1461 "cp858",
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1462 "cp860",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1463 "cp861",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1464 "cp862",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1465 "cp863",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1466 "cp864",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1467 "cp865",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1468 "cp866",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1469 "cp869",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1470 "cp874",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1471 "cp875",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1472 "cp932",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1473 "cp949",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1474 "cp950",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1475 "euc_jis_2004",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1476 "euc_jisx0213",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1477 "euc_jp",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1478 "euc_kr",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1479 "gb18030",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1480 "gb2312",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1481 "gbk",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1482 "hp_roman8",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1483 "hz",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1484 "idna",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1485 "iso2022_jp",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1486 "iso2022_jp_1",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1487 "iso2022_jp_2",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1488 "iso2022_jp_2004",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1489 "iso2022_jp_3",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1490 "iso2022_jp_ext",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1491 "iso2022_kr",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1492 "iso8859_1",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1493 "iso8859_10",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1494 "iso8859_11",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1495 "iso8859_13",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1496 "iso8859_14",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1497 "iso8859_15",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1498 "iso8859_16",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1499 "iso8859_2",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1500 "iso8859_3",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1501 "iso8859_4",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1502 "iso8859_5",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1503 "iso8859_6",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1504 "iso8859_7",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1505 "iso8859_8",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1506 "iso8859_9",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1507 "johab",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1508 "koi8_r",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1509 "koi8_u",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1510 "latin_1",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1511 "mac_cyrillic",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1512 "mac_greek",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1513 "mac_iceland",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1514 "mac_latin2",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1515 "mac_roman",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1516 "mac_turkish",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1517 "palmos",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1518 "ptcp154",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1519 "punycode",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1520 "raw_unicode_escape",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1521 "shift_jis",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1522 "shift_jis_2004",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1523 "shift_jisx0213",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1524 "tis_620",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1525 "unicode_escape",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1526 "unicode_internal",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1527 "utf_16",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1528 "utf_16_be",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1529 "utf_16_le",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1530 "utf_7",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1531 "utf_8",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1532 ]
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1533
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1534 if hasattr(codecs, "mbcs_encode"):
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1535 all_unicode_encodings.append("mbcs")
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1536
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1537 # The following encoding is not tested, because it's not supposed
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1538 # to work:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1539 # "undefined"
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1540
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1541 # The following encodings don't work in stateful mode
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1542 broken_unicode_with_streams = [
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1543 "punycode",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1544 "unicode_internal"
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1545 ]
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	1546 broken_incremental_coders = broken_unicode_with_streams + [
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	1547 "idna",
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	1548 ]
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1549
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	1550 class BasicUnicodeTest(unittest.TestCase, MixInCheckStateHandling):
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1551 def test_basics(self):
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1552 s = "abc123" # all codecs should be able to encode these
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1553 for encoding in all_unicode_encodings:
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1554 name = codecs.lookup(encoding).name
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1555 if encoding.endswith("_codec"):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1556 name += "_codec"
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1557 elif encoding == "latin_1":
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1558 name = "latin_1"
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1559 self.assertEqual(encoding.replace("_", "-"), name.replace("_", "-"))
73579 ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	1560
73600 fe2be7d35660 #13406: fix more deprecation warnings and move the deprecation of unicode-internal earlier in the code. Ezio Melotti <ezio.melotti@gmail.com> parents: 73589 diff changeset	1561 with support.check_warnings():
73579 ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	1562 # unicode-internal has been deprecated
ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	1563 (b, size) = codecs.getencoder(encoding)(s)
ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	1564 self.assertEqual(size, len(s), "%r != %r (encoding=%r)" % (size, len(s), encoding))
ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	1565 (chars, size) = codecs.getdecoder(encoding)(b)
ec8c0ac81706 "unicode_internal" codec has been deprecated: fix related tests Victor Stinner <victor.stinner@haypocalc.com> parents: 73336 diff changeset	1566 self.assertEqual(chars, s, "%r != %r (encoding=%r)" % (chars, s, encoding))
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1567
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1568 if encoding not in broken_unicode_with_streams:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1569 # check stream reader/writer
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1570 q = Queue(b"")
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1571 writer = codecs.getwriter(encoding)(q)
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1572 encodedresult = b""
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1573 for c in s:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1574 writer.write(c)
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43577 diff changeset	1575 chunk = q.read()
55593 0bb232078b8c Merged revisions 73715 via svnmerge from Georg Brandl <georg@python.org> parents: 54988 diff changeset	1576 self.assertTrue(type(chunk) is bytes, type(chunk))
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43577 diff changeset	1577 encodedresult += chunk
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1578 q = Queue(b"")
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1579 reader = codecs.getreader(encoding)(q)
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1580 decodedresult = ""
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1581 for c in encodedresult:
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1582 q.write(bytes([c]))
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1583 decodedresult += reader.read()
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1584 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1585
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1586 if encoding not in broken_incremental_coders:
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1587 # check incremental decoder/encoder (fetched via the Python
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1588 # and C API) and iterencode()/iterdecode()
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1589 try:
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1590 encoder = codecs.getincrementalencoder(encoding)()
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1591 cencoder = _testcapi.codec_incrementalencoder(encoding)
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1592 except LookupError: # no IncrementalEncoder
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1593 pass
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1594 else:
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1595 # check incremental decoder/encoder
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1596 encodedresult = b""
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1597 for c in s:
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1598 encodedresult += encoder.encode(c)
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1599 encodedresult += encoder.encode("", True)
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1600 decoder = codecs.getincrementaldecoder(encoding)()
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1601 decodedresult = ""
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1602 for c in encodedresult:
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1603 decodedresult += decoder.decode(bytes([c]))
41757 13a6c6bef609 Make test_codecs work. The CJK codecs now use bytes instead of str8 for Guido van Rossum <guido@python.org> parents: 41750 diff changeset	1604 decodedresult += decoder.decode(b"", True)
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1605 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1606
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1607 # check C API
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1608 encodedresult = b""
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1609 for c in s:
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1610 encodedresult += cencoder.encode(c)
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1611 encodedresult += cencoder.encode("", True)
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1612 cdecoder = _testcapi.codec_incrementaldecoder(encoding)
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1613 decodedresult = ""
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	1614 for c in encodedresult:
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1615 decodedresult += cdecoder.decode(bytes([c]))
41757 13a6c6bef609 Make test_codecs work. The CJK codecs now use bytes instead of str8 for Guido van Rossum <guido@python.org> parents: 41750 diff changeset	1616 decodedresult += cdecoder.decode(b"", True)
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1617 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1618
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1619 # check iterencode()/iterdecode()
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1620 result = "".join(codecs.iterdecode(codecs.iterencode(s, encoding), encoding))
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1621 self.assertEqual(result, s, "%r != %r (encoding=%r)" % (result, s, encoding))
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1622
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1623 # check iterencode()/iterdecode() with empty string
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1624 result = "".join(codecs.iterdecode(codecs.iterencode("", encoding), encoding))
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1625 self.assertEqual(result, "")
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1626
62367 283d9472df4e Issue #850997: mbcs encoding (Windows only) handles errors argument: strict Victor Stinner <victor.stinner@haypocalc.com> parents: 62256 diff changeset	1627 if encoding not in ("idna", "mbcs"):
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1628 # check incremental decoder/encoder with errors argument
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1629 try:
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1630 encoder = codecs.getincrementalencoder(encoding)("ignore")
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1631 cencoder = _testcapi.codec_incrementalencoder(encoding, "ignore")
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1632 except LookupError: # no IncrementalEncoder
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1633 pass
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1634 else:
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1635 encodedresult = b"".join(encoder.encode(c) for c in s)
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1636 decoder = codecs.getincrementaldecoder(encoding)("ignore")
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1637 decodedresult = "".join(decoder.decode(bytes([c])) for c in encodedresult)
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1638 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1639
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1640 encodedresult = b"".join(cencoder.encode(c) for c in s)
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1641 cdecoder = _testcapi.codec_incrementaldecoder(encoding, "ignore")
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1642 decodedresult = "".join(cdecoder.decode(bytes([c])) for c in encodedresult)
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1643 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1644
34750 679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1645 def test_seek(self):
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1646 # all codecs should be able to encode these
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1647 s = "%s\n%s\n" % (100"abc123", 100"def456")
34750 679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1648 for encoding in all_unicode_encodings:
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1649 if encoding == "idna": # FIXME: See SF bug #1163178
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1650 continue
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1651 if encoding in broken_unicode_with_streams:
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1652 continue
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	1653 reader = codecs.getreader(encoding)(io.BytesIO(s.encode(encoding)))
41642 a6eb6acfe04a Merged revisions 55007-55179 via svnmerge from Guido van Rossum <guido@python.org> parents: 41606 diff changeset	1654 for t in range(5):
34750 679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1655 # Test that calling seek resets the internal codec state and buffers
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1656 reader.seek(0, 0)
41757 13a6c6bef609 Make test_codecs work. The CJK codecs now use bytes instead of str8 for Guido van Rossum <guido@python.org> parents: 41750 diff changeset	1657 data = reader.read()
13a6c6bef609 Make test_codecs work. The CJK codecs now use bytes instead of str8 for Guido van Rossum <guido@python.org> parents: 41750 diff changeset	1658 self.assertEqual(s, data)
34750 679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1659
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1660 def test_bad_decode_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1661 for encoding in all_unicode_encodings:
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1662 decoder = codecs.getdecoder(encoding)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1663 self.assertRaises(TypeError, decoder)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1664 if encoding not in ("idna", "punycode"):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1665 self.assertRaises(TypeError, decoder, 42)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1666
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1667 def test_bad_encode_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1668 for encoding in all_unicode_encodings:
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1669 encoder = codecs.getencoder(encoding)
73600 fe2be7d35660 #13406: fix more deprecation warnings and move the deprecation of unicode-internal earlier in the code. Ezio Melotti <ezio.melotti@gmail.com> parents: 73589 diff changeset	1670 with support.check_warnings():
fe2be7d35660 #13406: fix more deprecation warnings and move the deprecation of unicode-internal earlier in the code. Ezio Melotti <ezio.melotti@gmail.com> parents: 73589 diff changeset	1671 # unicode-internal has been deprecated
fe2be7d35660 #13406: fix more deprecation warnings and move the deprecation of unicode-internal earlier in the code. Ezio Melotti <ezio.melotti@gmail.com> parents: 73589 diff changeset	1672 self.assertRaises(TypeError, encoder)
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1673
39442 654c380cf8b9 Merged revisions 46753-51188 via svnmerge from Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	1674 def test_encoding_map_type_initialized(self):
654c380cf8b9 Merged revisions 46753-51188 via svnmerge from Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	1675 from encodings import cp1140
654c380cf8b9 Merged revisions 46753-51188 via svnmerge from Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	1676 # This used to crash, we are only verifying there's no crash.
654c380cf8b9 Merged revisions 46753-51188 via svnmerge from Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	1677 table_type = type(cp1140.encoding_table)
654c380cf8b9 Merged revisions 46753-51188 via svnmerge from Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	1678 self.assertEqual(table_type, table_type)
654c380cf8b9 Merged revisions 46753-51188 via svnmerge from Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	1679
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	1680 def test_decoder_state(self):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	1681 # Check that getstate() and setstate() handle the state properly
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1682 u = "abc123"
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	1683 for encoding in all_unicode_encodings:
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	1684 if encoding not in broken_incremental_coders:
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	1685 self.check_state_handling_decode(encoding, u, u.encode(encoding))
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	1686 self.check_state_handling_encode(encoding, u, u.encode(encoding))
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40775 diff changeset	1687
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1688 class CharmapTest(unittest.TestCase):
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1689 def test_decode_with_string_map(self):
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1690 self.assertEqual(
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1691 codecs.charmap_decode(b"\x00\x01\x02", "strict", "abc"),
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1692 ("abc", 3)
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1693 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1694
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1695 self.assertEqual(
79108 c64dec45d46f Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78207 79107 diff changeset	1696 codecs.charmap_decode(b"\x00\x01\x02", "strict", "\U0010FFFFbc"),
c64dec45d46f Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78207 79107 diff changeset	1697 ("\U0010FFFFbc", 3)
c64dec45d46f Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78207 79107 diff changeset	1698 )
c64dec45d46f Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78207 79107 diff changeset	1699
79107 620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1700 self.assertRaises(UnicodeDecodeError,
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1701 codecs.charmap_decode, b"\x00\x01\x02", "strict", "ab"
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1702 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1703
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1704 self.assertEqual(
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1705 codecs.charmap_decode(b"\x00\x01\x02", "replace", "ab"),
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1706 ("ab\ufffd", 3)
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1707 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1708
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1709 self.assertEqual(
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1710 codecs.charmap_decode(b"\x00\x01\x02", "replace", "ab\ufffe"),
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1711 ("ab\ufffd", 3)
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1712 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1713
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1714 self.assertEqual(
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1715 codecs.charmap_decode(b"\x00\x01\x02", "ignore", "ab"),
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1716 ("ab", 3)
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1717 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1718
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1719 self.assertEqual(
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1720 codecs.charmap_decode(b"\x00\x01\x02", "ignore", "ab\ufffe"),
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1721 ("ab", 3)
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1722 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1723
41642 a6eb6acfe04a Merged revisions 55007-55179 via svnmerge from Guido van Rossum <guido@python.org> parents: 41606 diff changeset	1724 allbytes = bytes(range(256))
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1725 self.assertEqual(
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1726 codecs.charmap_decode(allbytes, "ignore", ""),
beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41473 diff changeset	1727 ("", len(allbytes))
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1728 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1729
79107 620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1730 def test_decode_with_int2str_map(self):
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1731 self.assertEqual(
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1732 codecs.charmap_decode(b"\x00\x01\x02", "strict",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1733 {0: 'a', 1: 'b', 2: 'c'}),
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1734 ("abc", 3)
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1735 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1736
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1737 self.assertEqual(
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1738 codecs.charmap_decode(b"\x00\x01\x02", "strict",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1739 {0: 'Aa', 1: 'Bb', 2: 'Cc'}),
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1740 ("AaBbCc", 3)
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1741 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1742
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1743 self.assertEqual(
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1744 codecs.charmap_decode(b"\x00\x01\x02", "strict",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1745 {0: '\U0010FFFF', 1: 'b', 2: 'c'}),
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1746 ("\U0010FFFFbc", 3)
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1747 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1748
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1749 self.assertEqual(
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1750 codecs.charmap_decode(b"\x00\x01\x02", "strict",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1751 {0: 'a', 1: 'b', 2: ''}),
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1752 ("ab", 3)
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1753 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1754
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1755 self.assertRaises(UnicodeDecodeError,
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1756 codecs.charmap_decode, b"\x00\x01\x02", "strict",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1757 {0: 'a', 1: 'b'}
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1758 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1759
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1760 self.assertEqual(
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1761 codecs.charmap_decode(b"\x00\x01\x02", "replace",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1762 {0: 'a', 1: 'b'}),
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1763 ("ab\ufffd", 3)
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1764 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1765
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1766 self.assertEqual(
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1767 codecs.charmap_decode(b"\x00\x01\x02", "replace",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1768 {0: 'a', 1: 'b', 2: None}),
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1769 ("ab\ufffd", 3)
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1770 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1771
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1772 self.assertEqual(
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1773 codecs.charmap_decode(b"\x00\x01\x02", "ignore",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1774 {0: 'a', 1: 'b'}),
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1775 ("ab", 3)
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1776 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1777
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1778 self.assertEqual(
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1779 codecs.charmap_decode(b"\x00\x01\x02", "ignore",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1780 {0: 'a', 1: 'b', 2: None}),
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1781 ("ab", 3)
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1782 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1783
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1784 allbytes = bytes(range(256))
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1785 self.assertEqual(
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1786 codecs.charmap_decode(allbytes, "ignore", {}),
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1787 ("", len(allbytes))
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1788 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1789
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1790 def test_decode_with_int2int_map(self):
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1791 a = ord('a')
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1792 b = ord('b')
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1793 c = ord('c')
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1794
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1795 self.assertEqual(
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1796 codecs.charmap_decode(b"\x00\x01\x02", "strict",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1797 {0: a, 1: b, 2: c}),
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1798 ("abc", 3)
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1799 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1800
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1801 # Issue #15379
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1802 self.assertEqual(
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1803 codecs.charmap_decode(b"\x00\x01\x02", "strict",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1804 {0: 0x10FFFF, 1: b, 2: c}),
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1805 ("\U0010FFFFbc", 3)
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1806 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1807
79108 c64dec45d46f Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78207 79107 diff changeset	1808 self.assertEqual(
c64dec45d46f Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78207 79107 diff changeset	1809 codecs.charmap_decode(b"\x00\x01\x02", "strict",
c64dec45d46f Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78207 79107 diff changeset	1810 {0: sys.maxunicode, 1: b, 2: c}),
c64dec45d46f Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78207 79107 diff changeset	1811 (chr(sys.maxunicode) + "bc", 3)
c64dec45d46f Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78207 79107 diff changeset	1812 )
c64dec45d46f Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78207 79107 diff changeset	1813
79107 620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1814 self.assertRaises(TypeError,
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1815 codecs.charmap_decode, b"\x00\x01\x02", "strict",
79108 c64dec45d46f Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78207 79107 diff changeset	1816 {0: sys.maxunicode + 1, 1: b, 2: c}
79107 620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1817 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1818
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1819 self.assertRaises(UnicodeDecodeError,
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1820 codecs.charmap_decode, b"\x00\x01\x02", "strict",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1821 {0: a, 1: b},
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1822 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1823
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1824 self.assertEqual(
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1825 codecs.charmap_decode(b"\x00\x01\x02", "replace",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1826 {0: a, 1: b}),
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1827 ("ab\ufffd", 3)
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1828 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1829
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1830 self.assertEqual(
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1831 codecs.charmap_decode(b"\x00\x01\x02", "ignore",
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1832 {0: a, 1: b}),
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1833 ("ab", 3)
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1834 )
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1835
620d23f7ad41 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78206 diff changeset	1836
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1837 class WithStmtTest(unittest.TestCase):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1838 def test_encodedfile(self):
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	1839 f = io.BytesIO(b"\xc3\xbc")
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1840 with codecs.EncodedFile(f, "latin-1", "utf-8") as ef:
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1841 self.assertEqual(ef.read(), b"\xfc")
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1842
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1843 def test_streamreaderwriter(self):
41681 1d334912a104 Use io.BytesIO instead of StringIO.StringIO in all Walter Dörwald <walter@livinglogic.de> parents: 41679 diff changeset	1844 f = io.BytesIO(b"\xc3\xbc")
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1845 info = codecs.lookup("utf-8")
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1846 with codecs.StreamReaderWriter(f, info.streamreader,
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1847 info.streamwriter, 'strict') as srw:
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1848 self.assertEqual(srw.read(), "\xfc")
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	1849
50975 ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	1850 class TypesTest(unittest.TestCase):
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	1851 def test_decode_unicode(self):
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	1852 # Most decoders don't accept unicode input
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	1853 decoders = [
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	1854 codecs.utf_7_decode,
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	1855 codecs.utf_8_decode,
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	1856 codecs.utf_16_le_decode,
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	1857 codecs.utf_16_be_decode,
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	1858 codecs.utf_16_ex_decode,
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	1859 codecs.utf_32_decode,
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	1860 codecs.utf_32_le_decode,
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	1861 codecs.utf_32_be_decode,
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	1862 codecs.utf_32_ex_decode,
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	1863 codecs.latin_1_decode,
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	1864 codecs.ascii_decode,
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	1865 codecs.charmap_decode,
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	1866 ]
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	1867 if hasattr(codecs, "mbcs_decode"):
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	1868 decoders.append(codecs.mbcs_decode)
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	1869 for decoder in decoders:
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	1870 self.assertRaises(TypeError, decoder, "xxx")
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	1871
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	1872 def test_unicode_escape(self):
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	1873 # Escape-decoding an unicode string is supported ang gives the same
ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	1874 # result as decoding the equivalent ASCII bytes string.
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1875 self.assertEqual(codecs.unicode_escape_decode(r"\u1234"), ("\u1234", 6))
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1876 self.assertEqual(codecs.unicode_escape_decode(br"\u1234"), ("\u1234", 6))
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1877 self.assertEqual(codecs.raw_unicode_escape_decode(r"\u1234"), ("\u1234", 6))
2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1878 self.assertEqual(codecs.raw_unicode_escape_decode(br"\u1234"), ("\u1234", 6))
50975 ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	1879
73906 bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	1880 self.assertRaises(UnicodeDecodeError, codecs.unicode_escape_decode, br"\U00110000")
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	1881 self.assertEqual(codecs.unicode_escape_decode(r"\U00110000", "replace"), ("\ufffd", 10))
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	1882
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	1883 self.assertRaises(UnicodeDecodeError, codecs.raw_unicode_escape_decode, br"\U00110000")
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	1884 self.assertEqual(codecs.raw_unicode_escape_decode(r"\U00110000", "replace"), ("\ufffd", 10))
bfa9d1ba36ae Write tests for invalid characters (U+00110000) Victor Stinner <victor.stinner@haypocalc.com> parents: 73600 diff changeset	1885
53966 7c29bfbfd352 Rename utf8b error handler to surrogateescape. Martin v. Löwis <martin@v.loewis.de> parents: 53965 diff changeset	1886 class SurrogateEscapeTest(unittest.TestCase):
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	1887
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	1888 def test_utf8(self):
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	1889 # Bad byte
53966 7c29bfbfd352 Rename utf8b error handler to surrogateescape. Martin v. Löwis <martin@v.loewis.de> parents: 53965 diff changeset	1890 self.assertEqual(b"foo\x80bar".decode("utf-8", "surrogateescape"),
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	1891 "foo\udc80bar")
53966 7c29bfbfd352 Rename utf8b error handler to surrogateescape. Martin v. Löwis <martin@v.loewis.de> parents: 53965 diff changeset	1892 self.assertEqual("foo\udc80bar".encode("utf-8", "surrogateescape"),
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	1893 b"foo\x80bar")
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	1894 # bad-utf-8 encoded surrogate
53966 7c29bfbfd352 Rename utf8b error handler to surrogateescape. Martin v. Löwis <martin@v.loewis.de> parents: 53965 diff changeset	1895 self.assertEqual(b"\xed\xb0\x80".decode("utf-8", "surrogateescape"),
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	1896 "\udced\udcb0\udc80")
53966 7c29bfbfd352 Rename utf8b error handler to surrogateescape. Martin v. Löwis <martin@v.loewis.de> parents: 53965 diff changeset	1897 self.assertEqual("\udced\udcb0\udc80".encode("utf-8", "surrogateescape"),
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	1898 b"\xed\xb0\x80")
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	1899
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	1900 def test_ascii(self):
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	1901 # bad byte
53966 7c29bfbfd352 Rename utf8b error handler to surrogateescape. Martin v. Löwis <martin@v.loewis.de> parents: 53965 diff changeset	1902 self.assertEqual(b"foo\x80bar".decode("ascii", "surrogateescape"),
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	1903 "foo\udc80bar")
53966 7c29bfbfd352 Rename utf8b error handler to surrogateescape. Martin v. Löwis <martin@v.loewis.de> parents: 53965 diff changeset	1904 self.assertEqual("foo\udc80bar".encode("ascii", "surrogateescape"),
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	1905 b"foo\x80bar")
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	1906
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	1907 def test_charmap(self):
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	1908 # bad byte: \xa5 is unmapped in iso-8859-3
53966 7c29bfbfd352 Rename utf8b error handler to surrogateescape. Martin v. Löwis <martin@v.loewis.de> parents: 53965 diff changeset	1909 self.assertEqual(b"foo\xa5bar".decode("iso-8859-3", "surrogateescape"),
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	1910 "foo\udca5bar")
53966 7c29bfbfd352 Rename utf8b error handler to surrogateescape. Martin v. Löwis <martin@v.loewis.de> parents: 53965 diff changeset	1911 self.assertEqual("foo\udca5bar".encode("iso-8859-3", "surrogateescape"),
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	1912 b"foo\xa5bar")
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	1913
54988 29fac8848489 Merged revisions 73698 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53966 diff changeset	1914 def test_latin1(self):
29fac8848489 Merged revisions 73698 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53966 diff changeset	1915 # Issue6373
68142 be6c38d1817b Normalize the encoding names for Latin-1 and UTF-8 to Marc-André Lemburg <mal@egenix.com> parents: 66837 diff changeset	1916 self.assertEqual("\udce4\udceb\udcef\udcf6\udcfc".encode("latin-1", "surrogateescape"),
54988 29fac8848489 Merged revisions 73698 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53966 diff changeset	1917 b"\xe4\xeb\xef\xf6\xfc")
29fac8848489 Merged revisions 73698 via svnmerge from Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 53966 diff changeset	1918
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1919
61850 846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	1920 class BomTest(unittest.TestCase):
846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	1921 def test_seek0(self):
846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	1922 data = "1234567890"
846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	1923 tests = ("utf-16",
846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	1924 "utf-16-le",
846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	1925 "utf-16-be",
846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	1926 "utf-32",
846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	1927 "utf-32-le",
846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	1928 "utf-32-be")
70292 b538445a9df8 test_codecs now removes the temporay file (created by the test) Victor Stinner <victor.stinner@haypocalc.com> parents: 66340 diff changeset	1929 self.addCleanup(support.unlink, support.TESTFN)
61850 846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	1930 for encoding in tests:
61863 cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1931 # Check if the BOM is written only once
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1932 with codecs.open(support.TESTFN, 'w+', encoding=encoding) as f:
61850 846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	1933 f.write(data)
846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	1934 f.write(data)
846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	1935 f.seek(0)
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1936 self.assertEqual(f.read(), data * 2)
61850 846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	1937 f.seek(0)
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1938 self.assertEqual(f.read(), data * 2)
61850 846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	1939
61863 cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1940 # Check that the BOM is written after a seek(0)
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1941 with codecs.open(support.TESTFN, 'w+', encoding=encoding) as f:
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1942 f.write(data[0])
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1943 self.assertNotEqual(f.tell(), 0)
61863 cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1944 f.seek(0)
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1945 f.write(data)
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1946 f.seek(0)
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1947 self.assertEqual(f.read(), data)
61863 cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1948
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1949 # (StreamWriter) Check that the BOM is written after a seek(0)
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1950 with codecs.open(support.TESTFN, 'w+', encoding=encoding) as f:
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1951 f.writer.write(data[0])
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1952 self.assertNotEqual(f.writer.tell(), 0)
61863 cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1953 f.writer.seek(0)
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1954 f.writer.write(data)
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1955 f.seek(0)
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1956 self.assertEqual(f.read(), data)
61863 cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1957
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1958 # Check that the BOM is not written after a seek() at a position
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1959 # different than the start
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1960 with codecs.open(support.TESTFN, 'w+', encoding=encoding) as f:
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1961 f.write(data)
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1962 f.seek(f.tell())
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1963 f.write(data)
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1964 f.seek(0)
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1965 self.assertEqual(f.read(), data * 2)
61863 cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1966
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1967 # (StreamWriter) Check that the BOM is not written after a seek()
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1968 # at a position different than the start
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1969 with codecs.open(support.TESTFN, 'w+', encoding=encoding) as f:
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1970 f.writer.write(data)
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1971 f.writer.seek(f.writer.tell())
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1972 f.writer.write(data)
cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1973 f.seek(0)
66340 2beb25369289 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62258 diff changeset	1974 self.assertEqual(f.read(), data * 2)
61863 cd4936691163 Merged revisions 81474 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61850 diff changeset	1975
61850 846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	1976
66605 7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	1977 bytes_transform_encodings = [
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	1978 "base64_codec",
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	1979 "uu_codec",
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	1980 "quopri_codec",
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	1981 "hex_codec",
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	1982 ]
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	1983 try:
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	1984 import zlib
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	1985 except ImportError:
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	1986 pass
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	1987 else:
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	1988 bytes_transform_encodings.append("zlib_codec")
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	1989 try:
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	1990 import bz2
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	1991 except ImportError:
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	1992 pass
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	1993 else:
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	1994 bytes_transform_encodings.append("bz2_codec")
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	1995
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	1996 class TransformCodecTest(unittest.TestCase):
66837 a811f7c10e82 remove (un)transform methods Benjamin Peterson <benjamin@python.org> parents: 66797 diff changeset	1997
66605 7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	1998 def test_basics(self):
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	1999 binput = bytes(range(256))
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2000 for encoding in bytes_transform_encodings:
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2001 # generic codecs interface
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2002 (o, size) = codecs.getencoder(encoding)(binput)
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2003 self.assertEqual(size, len(binput))
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2004 (i, size) = codecs.getdecoder(encoding)(o)
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2005 self.assertEqual(size, len(o))
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2006 self.assertEqual(i, binput)
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2007
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2008 def test_read(self):
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2009 for encoding in bytes_transform_encodings:
66837 a811f7c10e82 remove (un)transform methods Benjamin Peterson <benjamin@python.org> parents: 66797 diff changeset	2010 sin = codecs.encode(b"\x80", encoding)
66605 7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2011 reader = codecs.getreader(encoding)(io.BytesIO(sin))
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2012 sout = reader.read()
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2013 self.assertEqual(sout, b"\x80")
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2014
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2015 def test_readline(self):
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2016 for encoding in bytes_transform_encodings:
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2017 if encoding in ['uu_codec', 'zlib_codec']:
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2018 continue
66837 a811f7c10e82 remove (un)transform methods Benjamin Peterson <benjamin@python.org> parents: 66797 diff changeset	2019 sin = codecs.encode(b"\x80", encoding)
66605 7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2020 reader = codecs.getreader(encoding)(io.BytesIO(sin))
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2021 sout = reader.readline()
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2022 self.assertEqual(sout, b"\x80")
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2023
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2024
72977 5841920d1ef6 Issue #12281: Skip code page tests on non-Windows platforms Victor Stinner <victor.stinner@haypocalc.com> parents: 72976 diff changeset	2025 @unittest.skipUnless(sys.platform == 'win32',
5841920d1ef6 Issue #12281: Skip code page tests on non-Windows platforms Victor Stinner <victor.stinner@haypocalc.com> parents: 72976 diff changeset	2026 'code pages are specific to Windows')
72976 af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2027 class CodePageTest(unittest.TestCase):
73138 2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	2028 # CP_UTF8 is already tested by CP65001Test
72976 af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2029 CP_UTF8 = 65001
72977 5841920d1ef6 Issue #12281: Skip code page tests on non-Windows platforms Victor Stinner <victor.stinner@haypocalc.com> parents: 72976 diff changeset	2030
72976 af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2031 def test_invalid_code_page(self):
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2032 self.assertRaises(ValueError, codecs.code_page_encode, -1, 'a')
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2033 self.assertRaises(ValueError, codecs.code_page_decode, -1, b'a')
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2034 self.assertRaises(WindowsError, codecs.code_page_encode, 123, 'a')
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2035 self.assertRaises(WindowsError, codecs.code_page_decode, 123, b'a')
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2036
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2037 def test_code_page_name(self):
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2038 self.assertRaisesRegex(UnicodeEncodeError, 'cp932',
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2039 codecs.code_page_encode, 932, '\xff')
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2040 self.assertRaisesRegex(UnicodeDecodeError, 'cp932',
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2041 codecs.code_page_decode, 932, b'\x81\x00')
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2042 self.assertRaisesRegex(UnicodeDecodeError, 'CP_UTF8',
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2043 codecs.code_page_decode, self.CP_UTF8, b'\xff')
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2044
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2045 def check_decode(self, cp, tests):
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2046 for raw, errors, expected in tests:
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2047 if expected is not None:
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2048 try:
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2049 decoded = codecs.code_page_decode(cp, raw, errors)
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2050 except UnicodeDecodeError as err:
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2051 self.fail('Unable to decode %a from "cp%s" with '
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2052 'errors=%r: %s' % (raw, cp, errors, err))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2053 self.assertEqual(decoded[0], expected,
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2054 '%a.decode("cp%s", %r)=%a != %a'
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2055 % (raw, cp, errors, decoded[0], expected))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2056 # assert 0 <= decoded[1] <= len(raw)
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2057 self.assertGreaterEqual(decoded[1], 0)
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2058 self.assertLessEqual(decoded[1], len(raw))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2059 else:
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2060 self.assertRaises(UnicodeDecodeError,
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2061 codecs.code_page_decode, cp, raw, errors)
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2062
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2063 def check_encode(self, cp, tests):
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2064 for text, errors, expected in tests:
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2065 if expected is not None:
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2066 try:
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2067 encoded = codecs.code_page_encode(cp, text, errors)
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2068 except UnicodeEncodeError as err:
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2069 self.fail('Unable to encode %a to "cp%s" with '
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2070 'errors=%r: %s' % (text, cp, errors, err))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2071 self.assertEqual(encoded[0], expected,
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2072 '%a.encode("cp%s", %r)=%a != %a'
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2073 % (text, cp, errors, encoded[0], expected))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2074 self.assertEqual(encoded[1], len(text))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2075 else:
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2076 self.assertRaises(UnicodeEncodeError,
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2077 codecs.code_page_encode, cp, text, errors)
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2078
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2079 def test_cp932(self):
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2080 self.check_encode(932, (
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2081 ('abc', 'strict', b'abc'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2082 ('\uff44\u9a3e', 'strict', b'\x82\x84\xe9\x80'),
73138 2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	2083 # test error handlers
72976 af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2084 ('\xff', 'strict', None),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2085 ('[\xff]', 'ignore', b'[]'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2086 ('[\xff]', 'replace', b'[y]'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2087 ('[\u20ac]', 'replace', b'[?]'),
73138 2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	2088 ('[\xff]', 'backslashreplace', b'[\\xff]'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	2089 ('[\xff]', 'xmlcharrefreplace', b'[ÿ]'),
72976 af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2090 ))
72978 413b89242766 Issue #12281: Fix test_codecs.test_cp932() on Windows XP Victor Stinner <victor.stinner@haypocalc.com> parents: 72977 diff changeset	2091 self.check_decode(932, (
72976 af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2092 (b'abc', 'strict', 'abc'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2093 (b'\x82\x84\xe9\x80', 'strict', '\uff44\u9a3e'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2094 # invalid bytes
73138 2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	2095 (b'[\xff]', 'strict', None),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	2096 (b'[\xff]', 'ignore', '[]'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	2097 (b'[\xff]', 'replace', '[\ufffd]'),
2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	2098 (b'[\xff]', 'surrogateescape', '[\udcff]'),
72976 af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2099 (b'\x81\x00abc', 'strict', None),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2100 (b'\x81\x00abc', 'ignore', '\x00abc'),
72978 413b89242766 Issue #12281: Fix test_codecs.test_cp932() on Windows XP Victor Stinner <victor.stinner@haypocalc.com> parents: 72977 diff changeset	2101 (b'\x81\x00abc', 'replace', '\ufffd\x00abc'),
413b89242766 Issue #12281: Fix test_codecs.test_cp932() on Windows XP Victor Stinner <victor.stinner@haypocalc.com> parents: 72977 diff changeset	2102 ))
72976 af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2103
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2104 def test_cp1252(self):
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2105 self.check_encode(1252, (
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2106 ('abc', 'strict', b'abc'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2107 ('\xe9\u20ac', 'strict', b'\xe9\x80'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2108 ('\xff', 'strict', b'\xff'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2109 ('\u0141', 'strict', None),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2110 ('\u0141', 'ignore', b''),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2111 ('\u0141', 'replace', b'L'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2112 ))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2113 self.check_decode(1252, (
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2114 (b'abc', 'strict', 'abc'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2115 (b'\xe9\x80', 'strict', '\xe9\u20ac'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2116 (b'\xff', 'strict', '\xff'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2117 ))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2118
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2119 def test_cp_utf7(self):
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2120 cp = 65000
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2121 self.check_encode(cp, (
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2122 ('abc', 'strict', b'abc'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2123 ('\xe9\u20ac', 'strict', b'+AOkgrA-'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2124 ('\U0010ffff', 'strict', b'+2//f/w-'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2125 ('\udc80', 'strict', b'+3IA-'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2126 ('\ufffd', 'strict', b'+//0-'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2127 ))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2128 self.check_decode(cp, (
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2129 (b'abc', 'strict', 'abc'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2130 (b'+AOkgrA-', 'strict', '\xe9\u20ac'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2131 (b'+2//f/w-', 'strict', '\U0010ffff'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2132 (b'+3IA-', 'strict', '\udc80'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2133 (b'+//0-', 'strict', '\ufffd'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2134 # invalid bytes
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2135 (b'[+/]', 'strict', '[]'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2136 (b'[\xff]', 'strict', '[\xff]'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2137 ))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2138
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2139 def test_multibyte_encoding(self):
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2140 self.check_decode(932, (
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2141 (b'\x84\xe9\x80', 'ignore', '\u9a3e'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2142 (b'\x84\xe9\x80', 'replace', '\ufffd\u9a3e'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2143 ))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2144 self.check_decode(self.CP_UTF8, (
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2145 (b'\xff\xf4\x8f\xbf\xbf', 'ignore', '\U0010ffff'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2146 (b'\xff\xf4\x8f\xbf\xbf', 'replace', '\ufffd\U0010ffff'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2147 ))
73138 2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	2148 if VISTA_OR_LATER:
72976 af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2149 self.check_encode(self.CP_UTF8, (
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2150 ('[\U0010ffff\uDC80]', 'ignore', b'[\xf4\x8f\xbf\xbf]'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2151 ('[\U0010ffff\uDC80]', 'replace', b'[\xf4\x8f\xbf\xbf?]'),
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2152 ))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2153
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2154 def test_incremental(self):
73336 17341b93871b Cleanup decode_code_page_stateful() and encode_code_page() Victor Stinner <victor.stinner@haypocalc.com> parents: 73138 diff changeset	2155 decoded = codecs.code_page_decode(932, b'\x82', 'strict', False)
17341b93871b Cleanup decode_code_page_stateful() and encode_code_page() Victor Stinner <victor.stinner@haypocalc.com> parents: 73138 diff changeset	2156 self.assertEqual(decoded, ('', 0))
17341b93871b Cleanup decode_code_page_stateful() and encode_code_page() Victor Stinner <victor.stinner@haypocalc.com> parents: 73138 diff changeset	2157
72976 af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2158 decoded = codecs.code_page_decode(932,
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2159 b'\xe9\x80\xe9', 'strict',
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2160 False)
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2161 self.assertEqual(decoded, ('\u9a3e', 2))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2162
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2163 decoded = codecs.code_page_decode(932,
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2164 b'\xe9\x80\xe9\x80', 'strict',
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2165 False)
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2166 self.assertEqual(decoded, ('\u9a3e\u9a3e', 4))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2167
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2168 decoded = codecs.code_page_decode(932,
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2169 b'abc', 'strict',
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2170 False)
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2171 self.assertEqual(decoded, ('abc', 3))
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2172
af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2173
20013 e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	2174 def test_main():
47201 9c930a32cebb #2621 rename test.test_support to test.support Benjamin Peterson <benjamin@python.org> parents: 43933 diff changeset	2175 support.run_unittest(
42626 f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	2176 UTF32Test,
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	2177 UTF32LETest,
f1a2d51f1cab Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and Walter Dörwald <walter@livinglogic.de> parents: 42124 diff changeset	2178 UTF32BETest,
28917 1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	2179 UTF16Test,
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	2180 UTF16LETest,
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	2181 UTF16BETest,
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	2182 UTF8Test,
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	2183 UTF8SigTest,
73138 2cad20e2e588 Close #13247: Add cp65001 codec, the Windows UTF-8 (CP_UTF8) Victor Stinner <victor.stinner@haypocalc.com> parents: 72978 diff changeset	2184 CP65001Test,
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	2185 UTF7Test,
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	2186 UTF16ExTest,
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	2187 ReadBufferTest,
28917 1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	2188 RecodingTest,
1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	2189 PunycodeTest,
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	2190 UnicodeInternalTest,
31795 bf37429f6d9f Add test case for unicode(somestring, "idna"). Martin v. Löwis <martin@v.loewis.de> parents: 28917 diff changeset	2191 NameprepTest,
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	2192 IDNACodecTest,
33782 a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	2193 CodecsModuleTest,
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	2194 StreamReaderTest,
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	2195 EncodedFileTest,
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	2196 BasicUnicodeTest,
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	2197 CharmapTest,
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 39442 diff changeset	2198 WithStmtTest,
50975 ec6f29608bf0 Merged revisions 68855 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 48782 diff changeset	2199 TypesTest,
53966 7c29bfbfd352 Rename utf8b error handler to surrogateescape. Martin v. Löwis <martin@v.loewis.de> parents: 53965 diff changeset	2200 SurrogateEscapeTest,
61850 846dbcf970ab Merged revisions 81461 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 60875 diff changeset	2201 BomTest,
66605 7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 66310 diff changeset	2202 TransformCodecTest,
72976 af0800b986b7 Issue #12281: Rewrite the MBCS codec to handle correctly replace and ignore Victor Stinner <victor.stinner@haypocalc.com> parents: 72683 diff changeset	2203 CodePageTest,
28917 1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	2204 )
20013 e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	2205
e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	2206
e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	2207 if __name__ == "__main__":
e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	2208 test_main()

Mercurial > cpython-lru_cache-weakref

annotate Lib/test/test_codecs.py @ 79691:05c8928362ee