cpython-lru_cache-weakref: Lib/test/test

annotate Lib/test/test_codecs.py @ 80661:94de77bd0b4b 2.7

Issue #16194: document imp.load_dynamic problems when read several extension modules from the same binary object file.

author	Andrew Svetlov <andrew.svetlov@gmail.com>
date	Thu, 29 Nov 2012 19:51:28 +0200
parents	c7ce91756472
children	f2353e74b335

rev	line source
24364 aa0d2d51fbde Get rid of relative imports in all unittests. Now anything that Barry Warsaw <barry@python.org> parents: 20013 diff changeset	1 from test import test_support
aa0d2d51fbde Get rid of relative imports in all unittests. Now anything that Barry Warsaw <barry@python.org> parents: 20013 diff changeset	2 import unittest
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	3 import codecs
71492 739958134fe5 Issue #1813: Fix codec lookup and setting/getting locales under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70295 diff changeset	4 import locale
37077 38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	5 import sys, StringIO, _testcapi
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	6
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	7 class Queue(object):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	8 """
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	9 queue: write bytes at one end, read bytes from the other end
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	10 """
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	11 def __init__(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	12 self._buffer = ""
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	13
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	14 def write(self, chars):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	15 self._buffer += chars
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	16
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	17 def read(self, size=-1):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	18 if size<0:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	19 s = self._buffer
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	20 self._buffer = ""
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	21 return s
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	22 else:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	23 s = self._buffer[:size]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	24 self._buffer = self._buffer[size:]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	25 return s
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	26
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	27 class ReadTest(unittest.TestCase):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	28 def check_partial(self, input, partialresults):
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	29 # get a StreamReader for the encoding and feed the bytestring version
43904 49681d436df9 Backport r59047: Fix typo in comment. Walter Dörwald <walter@livinglogic.de> parents: 41533 diff changeset	30 # of input to the reader byte by byte. Read everything available from
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	31 # the StreamReader and check that the results equal the appropriate
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	32 # entries from partialresults.
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	33 q = Queue()
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	34 r = codecs.getreader(self.encoding)(q)
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	35 result = u""
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	36 for (c, partialresult) in zip(input.encode(self.encoding), partialresults):
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	37 q.write(c)
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	38 result += r.read()
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	39 self.assertEqual(result, partialresult)
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	40 # check that there's nothing left in the buffers
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	41 self.assertEqual(r.read(), u"")
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	42 self.assertEqual(r.bytebuffer, "")
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	43 self.assertEqual(r.charbuffer, u"")
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	44
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	45 # do the check again, this time using a incremental decoder
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	46 d = codecs.getincrementaldecoder(self.encoding)()
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	47 result = u""
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	48 for (c, partialresult) in zip(input.encode(self.encoding), partialresults):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	49 result += d.decode(c)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	50 self.assertEqual(result, partialresult)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	51 # check that there's nothing left in the buffers
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	52 self.assertEqual(d.decode("", True), u"")
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	53 self.assertEqual(d.buffer, "")
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	54
43932 e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	55 # Check whether the reset method works properly
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	56 d.reset()
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	57 result = u""
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	58 for (c, partialresult) in zip(input.encode(self.encoding), partialresults):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	59 result += d.decode(c)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	60 self.assertEqual(result, partialresult)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	61 # check that there's nothing left in the buffers
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	62 self.assertEqual(d.decode("", True), u"")
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	63 self.assertEqual(d.buffer, "")
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	64
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	65 # check iterdecode()
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	66 encoded = input.encode(self.encoding)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	67 self.assertEqual(
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	68 input,
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	69 u"".join(codecs.iterdecode(encoded, self.encoding))
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	70 )
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	71
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	72 def test_readline(self):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	73 def getreader(input):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	74 stream = StringIO.StringIO(input.encode(self.encoding))
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	75 return codecs.getreader(self.encoding)(stream)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	76
36855 204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	77 def readalllines(input, keepends=True, size=None):
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	78 reader = getreader(input)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	79 lines = []
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	80 while True:
36855 204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	81 line = reader.readline(size=size, keepends=keepends)
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	82 if not line:
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	83 break
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	84 lines.append(line)
36855 204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	85 return "\|".join(lines)
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	86
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	87 s = u"foo\nbar\r\nbaz\rspam\u2028eggs"
36855 204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	88 sexpected = u"foo\n\|bar\r\n\|baz\r\|spam\u2028\|eggs"
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	89 sexpectednoends = u"foo\|bar\|baz\|spam\|eggs"
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	90 self.assertEqual(readalllines(s, True), sexpected)
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	91 self.assertEqual(readalllines(s, False), sexpectednoends)
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	92 self.assertEqual(readalllines(s, True, 10), sexpected)
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	93 self.assertEqual(readalllines(s, False, 10), sexpectednoends)
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	94
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	95 # Test long lines (multiple calls to read() in readline())
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	96 vw = []
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	97 vwo = []
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	98 for (i, lineend) in enumerate(u"\n \r\n \r \u2028".split()):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	99 vw.append((i200)u"\3042" + lineend)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	100 vwo.append((i200)u"\3042")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	101 self.assertEqual(readalllines("".join(vw), True), "".join(vw))
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	102 self.assertEqual(readalllines("".join(vw), False),"".join(vwo))
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	103
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	104 # Test lines where the first read might end with \r, so the
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	105 # reader has to look ahead whether this is a lone \r or a \r\n
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	106 for size in xrange(80):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	107 for lineend in u"\n \r\n \r \u2028".split():
34865 b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	108 s = 10(sizeu"a" + lineend + u"xxx\n")
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	109 reader = getreader(s)
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	110 for i in xrange(10):
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	111 self.assertEqual(
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	112 reader.readline(keepends=True),
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	113 size*u"a" + lineend,
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	114 )
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	115 reader = getreader(s)
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	116 for i in xrange(10):
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	117 self.assertEqual(
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	118 reader.readline(keepends=False),
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	119 size*u"a",
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	120 )
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	121
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	122 def test_bug1175396(self):
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	123 s = [
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	124 '<%!--===================================================\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	125 ' BLOG index page: show recent articles,\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	126 ' today\'s articles, or articles of a specific date.\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	127 '========================================================--%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	128 '<%@inputencoding="ISO-8859-1"%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	129 '<%@pagetemplate=TEMPLATE.y%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	130 '<%@import=import frog.util, frog%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	131 '<%@import=import frog.objects%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	132 '<%@import=from frog.storageerrors import StorageError%>\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	133 '<%\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	134 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	135 'import logging\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	136 'log=logging.getLogger("Snakelets.logger")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	137 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	138 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	139 'user=self.SessionCtx.user\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	140 'storageEngine=self.SessionCtx.storageEngine\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	141 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	142 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	143 'def readArticlesFromDate(date, count=None):\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	144 ' entryids=storageEngine.listBlogEntries(date)\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	145 ' entryids.reverse() # descending\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	146 ' if count:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	147 ' entryids=entryids[:count]\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	148 ' try:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	149 ' return [ frog.objects.BlogEntry.load(storageEngine, date, Id) for Id in entryids ]\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	150 ' except StorageError,x:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	151 ' log.error("Error loading articles: "+str(x))\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	152 ' self.abort("cannot load articles")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	153 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	154 'showdate=None\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	155 '\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	156 'arg=self.Request.getArg()\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	157 'if arg=="today":\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	158 ' #-------------------- TODAY\'S ARTICLES\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	159 ' self.write("<h2>Today\'s articles</h2>")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	160 ' showdate = frog.util.isodatestr() \r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	161 ' entries = readArticlesFromDate(showdate)\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	162 'elif arg=="active":\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	163 ' #-------------------- ACTIVE ARTICLES redirect\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	164 ' self.Yredirect("active.y")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	165 'elif arg=="login":\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	166 ' #-------------------- LOGIN PAGE redirect\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	167 ' self.Yredirect("login.y")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	168 'elif arg=="date":\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	169 ' #-------------------- ARTICLES OF A SPECIFIC DATE\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	170 ' showdate = self.Request.getParameter("date")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	171 ' self.write("<h2>Articles written on %s</h2>"% frog.util.mediumdatestr(showdate))\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	172 ' entries = readArticlesFromDate(showdate)\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	173 'else:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	174 ' #-------------------- RECENT ARTICLES\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	175 ' self.write("<h2>Recent articles</h2>")\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	176 ' dates=storageEngine.listBlogEntryDates()\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	177 ' if dates:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	178 ' entries=[]\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	179 ' SHOWAMOUNT=10\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	180 ' for showdate in dates:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	181 ' entries.extend( readArticlesFromDate(showdate, SHOWAMOUNT-len(entries)) )\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	182 ' if len(entries)>=SHOWAMOUNT:\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	183 ' break\r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	184 ' \r\n',
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	185 ]
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	186 stream = StringIO.StringIO("".join(s).encode(self.encoding))
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	187 reader = codecs.getreader(self.encoding)(stream)
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	188 for (i, line) in enumerate(reader):
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34750 diff changeset	189 self.assertEqual(line, s[i])
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	190
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	191 def test_readlinequeue(self):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	192 q = Queue()
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	193 writer = codecs.getwriter(self.encoding)(q)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	194 reader = codecs.getreader(self.encoding)(q)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	195
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	196 # No lineends
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	197 writer.write(u"foo\r")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	198 self.assertEqual(reader.readline(keepends=False), u"foo")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	199 writer.write(u"\nbar\r")
34925 1331c7799640 Update test to the current readline() behaviour. Walter Dörwald <walter@livinglogic.de> parents: 34865 diff changeset	200 self.assertEqual(reader.readline(keepends=False), u"")
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	201 self.assertEqual(reader.readline(keepends=False), u"bar")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	202 writer.write(u"baz")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	203 self.assertEqual(reader.readline(keepends=False), u"baz")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	204 self.assertEqual(reader.readline(keepends=False), u"")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	205
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	206 # Lineends
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	207 writer.write(u"foo\r")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	208 self.assertEqual(reader.readline(keepends=True), u"foo\r")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	209 writer.write(u"\nbar\r")
34925 1331c7799640 Update test to the current readline() behaviour. Walter Dörwald <walter@livinglogic.de> parents: 34865 diff changeset	210 self.assertEqual(reader.readline(keepends=True), u"\n")
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	211 self.assertEqual(reader.readline(keepends=True), u"bar\r")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	212 writer.write(u"baz")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	213 self.assertEqual(reader.readline(keepends=True), u"baz")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	214 self.assertEqual(reader.readline(keepends=True), u"")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	215 writer.write(u"foo\r\n")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	216 self.assertEqual(reader.readline(keepends=True), u"foo\r\n")
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	217
34384 a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	218 def test_bug1098990_a(self):
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	219 s1 = u"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	220 s2 = u"offending line: ladfj askldfj klasdj fskla dfzaskdj fasklfj laskd fjasklfzzzzaa%whereisthis!!!\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	221 s3 = u"next line.\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	222
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	223 s = (s1+s2+s3).encode(self.encoding)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	224 stream = StringIO.StringIO(s)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	225 reader = codecs.getreader(self.encoding)(stream)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	226 self.assertEqual(reader.readline(), s1)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	227 self.assertEqual(reader.readline(), s2)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	228 self.assertEqual(reader.readline(), s3)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	229 self.assertEqual(reader.readline(), u"")
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	230
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	231 def test_bug1098990_b(self):
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	232 s1 = u"aaaaaaaaaaaaaaaaaaaaaaaa\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	233 s2 = u"bbbbbbbbbbbbbbbbbbbbbbbb\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	234 s3 = u"stillokay:bbbbxx\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	235 s4 = u"broken!!!!badbad\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	236 s5 = u"againokay.\r\n"
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	237
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	238 s = (s1+s2+s3+s4+s5).encode(self.encoding)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	239 stream = StringIO.StringIO(s)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	240 reader = codecs.getreader(self.encoding)(stream)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	241 self.assertEqual(reader.readline(), s1)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	242 self.assertEqual(reader.readline(), s2)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	243 self.assertEqual(reader.readline(), s3)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	244 self.assertEqual(reader.readline(), s4)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	245 self.assertEqual(reader.readline(), s5)
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	246 self.assertEqual(reader.readline(), u"")
a216003cef04 Fix and test for SF bug #1098990: codec readline() splits lines apart. Walter Dörwald <walter@livinglogic.de> parents: 34279 diff changeset	247
42649 c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	248 class UTF32Test(ReadTest):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	249 encoding = "utf-32"
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	250
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	251 spamle = ('\xff\xfe\x00\x00'
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	252 's\x00\x00\x00p\x00\x00\x00a\x00\x00\x00m\x00\x00\x00'
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	253 's\x00\x00\x00p\x00\x00\x00a\x00\x00\x00m\x00\x00\x00')
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	254 spambe = ('\x00\x00\xfe\xff'
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	255 '\x00\x00\x00s\x00\x00\x00p\x00\x00\x00a\x00\x00\x00m'
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	256 '\x00\x00\x00s\x00\x00\x00p\x00\x00\x00a\x00\x00\x00m')
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	257
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	258 def test_only_one_bom(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	259 _,_,reader,writer = codecs.lookup(self.encoding)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	260 # encode some stream
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	261 s = StringIO.StringIO()
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	262 f = writer(s)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	263 f.write(u"spam")
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	264 f.write(u"spam")
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	265 d = s.getvalue()
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	266 # check whether there is exactly one BOM in it
55002 6768b2350c57 convert usage of fail* to assert* Benjamin Peterson <benjamin@python.org> parents: 53855 diff changeset	267 self.assertTrue(d == self.spamle or d == self.spambe)
42649 c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	268 # try to read it back
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	269 s = StringIO.StringIO(d)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	270 f = reader(s)
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	271 self.assertEqual(f.read(), u"spamspam")
42649 c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	272
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	273 def test_badbom(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	274 s = StringIO.StringIO(4*"\xff")
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	275 f = codecs.getreader(self.encoding)(s)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	276 self.assertRaises(UnicodeError, f.read)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	277
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	278 s = StringIO.StringIO(8*"\xff")
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	279 f = codecs.getreader(self.encoding)(s)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	280 self.assertRaises(UnicodeError, f.read)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	281
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	282 def test_partial(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	283 self.check_partial(
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	284 u"\x00\xff\u0100\uffff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	285 [
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	286 u"", # first byte of BOM read
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	287 u"", # second byte of BOM read
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	288 u"", # third byte of BOM read
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	289 u"", # fourth byte of BOM read => byteorder known
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	290 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	291 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	292 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	293 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	294 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	295 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	296 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	297 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	298 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	299 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	300 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	301 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	302 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	303 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	304 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	305 u"\x00\xff\u0100\uffff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	306 ]
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	307 )
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	308
56034 0ff37e200292 Merged revisions 74869 via svnmerge from Georg Brandl <georg@python.org> parents: 43932 diff changeset	309 def test_handlers(self):
0ff37e200292 Merged revisions 74869 via svnmerge from Georg Brandl <georg@python.org> parents: 43932 diff changeset	310 self.assertEqual((u'\ufffd', 1),
0ff37e200292 Merged revisions 74869 via svnmerge from Georg Brandl <georg@python.org> parents: 43932 diff changeset	311 codecs.utf_32_decode('\x01', 'replace', True))
0ff37e200292 Merged revisions 74869 via svnmerge from Georg Brandl <georg@python.org> parents: 43932 diff changeset	312 self.assertEqual((u'', 1),
0ff37e200292 Merged revisions 74869 via svnmerge from Georg Brandl <georg@python.org> parents: 43932 diff changeset	313 codecs.utf_32_decode('\x01', 'ignore', True))
0ff37e200292 Merged revisions 74869 via svnmerge from Georg Brandl <georg@python.org> parents: 43932 diff changeset	314
42649 c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	315 def test_errors(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	316 self.assertRaises(UnicodeDecodeError, codecs.utf_32_decode,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	317 "\xff", "strict", True)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	318
62257 22c045d1bc8c Merged revisions 81907 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62214 diff changeset	319 def test_issue8941(self):
22c045d1bc8c Merged revisions 81907 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62214 diff changeset	320 # Issue #8941: insufficient result allocation when decoding into
22c045d1bc8c Merged revisions 81907 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62214 diff changeset	321 # surrogate pairs on UCS-2 builds.
22c045d1bc8c Merged revisions 81907 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62214 diff changeset	322 encoded_le = '\xff\xfe\x00\x00' + '\x00\x00\x01\x00' * 1024
22c045d1bc8c Merged revisions 81907 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62214 diff changeset	323 self.assertEqual(u'\U00010000' * 1024,
22c045d1bc8c Merged revisions 81907 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62214 diff changeset	324 codecs.utf_32_decode(encoded_le)[0])
22c045d1bc8c Merged revisions 81907 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62214 diff changeset	325 encoded_be = '\x00\x00\xfe\xff' + '\x00\x01\x00\x00' * 1024
22c045d1bc8c Merged revisions 81907 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62214 diff changeset	326 self.assertEqual(u'\U00010000' * 1024,
22c045d1bc8c Merged revisions 81907 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62214 diff changeset	327 codecs.utf_32_decode(encoded_be)[0])
22c045d1bc8c Merged revisions 81907 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62214 diff changeset	328
42649 c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	329 class UTF32LETest(ReadTest):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	330 encoding = "utf-32-le"
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	331
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	332 def test_partial(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	333 self.check_partial(
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	334 u"\x00\xff\u0100\uffff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	335 [
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	336 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	337 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	338 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	339 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	340 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	341 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	342 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	343 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	344 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	345 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	346 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	347 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	348 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	349 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	350 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	351 u"\x00\xff\u0100\uffff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	352 ]
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	353 )
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	354
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	355 def test_simple(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	356 self.assertEqual(u"\U00010203".encode(self.encoding), "\x03\x02\x01\x00")
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	357
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	358 def test_errors(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	359 self.assertRaises(UnicodeDecodeError, codecs.utf_32_le_decode,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	360 "\xff", "strict", True)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	361
62257 22c045d1bc8c Merged revisions 81907 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62214 diff changeset	362 def test_issue8941(self):
22c045d1bc8c Merged revisions 81907 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62214 diff changeset	363 # Issue #8941: insufficient result allocation when decoding into
22c045d1bc8c Merged revisions 81907 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62214 diff changeset	364 # surrogate pairs on UCS-2 builds.
22c045d1bc8c Merged revisions 81907 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62214 diff changeset	365 encoded = '\x00\x00\x01\x00' * 1024
22c045d1bc8c Merged revisions 81907 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62214 diff changeset	366 self.assertEqual(u'\U00010000' * 1024,
22c045d1bc8c Merged revisions 81907 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62214 diff changeset	367 codecs.utf_32_le_decode(encoded)[0])
22c045d1bc8c Merged revisions 81907 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62214 diff changeset	368
42649 c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	369 class UTF32BETest(ReadTest):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	370 encoding = "utf-32-be"
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	371
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	372 def test_partial(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	373 self.check_partial(
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	374 u"\x00\xff\u0100\uffff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	375 [
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	376 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	377 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	378 u"",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	379 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	380 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	381 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	382 u"\x00",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	383 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	384 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	385 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	386 u"\x00\xff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	387 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	388 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	389 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	390 u"\x00\xff\u0100",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	391 u"\x00\xff\u0100\uffff",
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	392 ]
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	393 )
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	394
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	395 def test_simple(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	396 self.assertEqual(u"\U00010203".encode(self.encoding), "\x00\x01\x02\x03")
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	397
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	398 def test_errors(self):
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	399 self.assertRaises(UnicodeDecodeError, codecs.utf_32_be_decode,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	400 "\xff", "strict", True)
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	401
62257 22c045d1bc8c Merged revisions 81907 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62214 diff changeset	402 def test_issue8941(self):
22c045d1bc8c Merged revisions 81907 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62214 diff changeset	403 # Issue #8941: insufficient result allocation when decoding into
22c045d1bc8c Merged revisions 81907 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62214 diff changeset	404 # surrogate pairs on UCS-2 builds.
22c045d1bc8c Merged revisions 81907 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62214 diff changeset	405 encoded = '\x00\x01\x00\x00' * 1024
22c045d1bc8c Merged revisions 81907 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62214 diff changeset	406 self.assertEqual(u'\U00010000' * 1024,
22c045d1bc8c Merged revisions 81907 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62214 diff changeset	407 codecs.utf_32_be_decode(encoded)[0])
22c045d1bc8c Merged revisions 81907 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62214 diff changeset	408
22c045d1bc8c Merged revisions 81907 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 62214 diff changeset	409
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	410 class UTF16Test(ReadTest):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	411 encoding = "utf-16"
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	412
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	413 spamle = '\xff\xfes\x00p\x00a\x00m\x00s\x00p\x00a\x00m\x00'
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	414 spambe = '\xfe\xff\x00s\x00p\x00a\x00m\x00s\x00p\x00a\x00m'
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	415
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	416 def test_only_one_bom(self):
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	417 _,_,reader,writer = codecs.lookup(self.encoding)
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	418 # encode some stream
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	419 s = StringIO.StringIO()
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	420 f = writer(s)
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	421 f.write(u"spam")
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	422 f.write(u"spam")
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	423 d = s.getvalue()
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	424 # check whether there is exactly one BOM in it
55002 6768b2350c57 convert usage of fail* to assert* Benjamin Peterson <benjamin@python.org> parents: 53855 diff changeset	425 self.assertTrue(d == self.spamle or d == self.spambe)
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	426 # try to read it back
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	427 s = StringIO.StringIO(d)
f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	428 f = reader(s)
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	429 self.assertEqual(f.read(), u"spamspam")
18163 f926eb1f5770 Test by Martin v. Loewis for the new UTF-16 codec handling of BOM Marc-André Lemburg <mal@egenix.com> parents: diff changeset	430
34538 37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	431 def test_badbom(self):
37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	432 s = StringIO.StringIO("\xff\xff")
34562 7a1277fad04e Fix stupid typo: Don't read from a writer. Walter Dörwald <walter@livinglogic.de> parents: 34538 diff changeset	433 f = codecs.getreader(self.encoding)(s)
34538 37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	434 self.assertRaises(UnicodeError, f.read)
37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	435
37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	436 s = StringIO.StringIO("\xff\xff\xff\xff")
34562 7a1277fad04e Fix stupid typo: Don't read from a writer. Walter Dörwald <walter@livinglogic.de> parents: 34538 diff changeset	437 f = codecs.getreader(self.encoding)(s)
34538 37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	438 self.assertRaises(UnicodeError, f.read)
37b2f11c1a00 Add a test for UTF-16 reading where the byte sequence doesn't start with Walter Dörwald <walter@livinglogic.de> parents: 34384 diff changeset	439
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	440 def test_partial(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	441 self.check_partial(
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	442 u"\x00\xff\u0100\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	443 [
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	444 u"", # first byte of BOM read
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	445 u"", # second byte of BOM read => byteorder known
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	446 u"",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	447 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	448 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	449 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	450 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	451 u"\x00\xff\u0100",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	452 u"\x00\xff\u0100",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	453 u"\x00\xff\u0100\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	454 ]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	455 )
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	456
56034 0ff37e200292 Merged revisions 74869 via svnmerge from Georg Brandl <georg@python.org> parents: 43932 diff changeset	457 def test_handlers(self):
0ff37e200292 Merged revisions 74869 via svnmerge from Georg Brandl <georg@python.org> parents: 43932 diff changeset	458 self.assertEqual((u'\ufffd', 1),
0ff37e200292 Merged revisions 74869 via svnmerge from Georg Brandl <georg@python.org> parents: 43932 diff changeset	459 codecs.utf_16_decode('\x01', 'replace', True))
0ff37e200292 Merged revisions 74869 via svnmerge from Georg Brandl <georg@python.org> parents: 43932 diff changeset	460 self.assertEqual((u'', 1),
0ff37e200292 Merged revisions 74869 via svnmerge from Georg Brandl <georg@python.org> parents: 43932 diff changeset	461 codecs.utf_16_decode('\x01', 'ignore', True))
0ff37e200292 Merged revisions 74869 via svnmerge from Georg Brandl <georg@python.org> parents: 43932 diff changeset	462
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	463 def test_errors(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	464 self.assertRaises(UnicodeDecodeError, codecs.utf_16_decode, "\xff", "strict", True)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	465
59169 4066b45d28ab Merged revisions 78461,78482 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 56034 diff changeset	466 def test_bug691291(self):
4066b45d28ab Merged revisions 78461,78482 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 56034 diff changeset	467 # Files are always opened in binary mode, even if no binary mode was
4066b45d28ab Merged revisions 78461,78482 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 56034 diff changeset	468 # specified. This means that no automatic conversion of '\n' is done
4066b45d28ab Merged revisions 78461,78482 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 56034 diff changeset	469 # on reading and writing.
4066b45d28ab Merged revisions 78461,78482 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 56034 diff changeset	470 s1 = u'Hello\r\nworld\r\n'
4066b45d28ab Merged revisions 78461,78482 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 56034 diff changeset	471
4066b45d28ab Merged revisions 78461,78482 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 56034 diff changeset	472 s = s1.encode(self.encoding)
70295 10c7fac542e6 test_codecs now removes the temporay file (created by the test) Victor Stinner <victor.stinner@haypocalc.com> parents: 68256 diff changeset	473 self.addCleanup(test_support.unlink, test_support.TESTFN)
10c7fac542e6 test_codecs now removes the temporay file (created by the test) Victor Stinner <victor.stinner@haypocalc.com> parents: 68256 diff changeset	474 with open(test_support.TESTFN, 'wb') as fp:
10c7fac542e6 test_codecs now removes the temporay file (created by the test) Victor Stinner <victor.stinner@haypocalc.com> parents: 68256 diff changeset	475 fp.write(s)
10c7fac542e6 test_codecs now removes the temporay file (created by the test) Victor Stinner <victor.stinner@haypocalc.com> parents: 68256 diff changeset	476 with codecs.open(test_support.TESTFN, 'U', encoding=self.encoding) as reader:
10c7fac542e6 test_codecs now removes the temporay file (created by the test) Victor Stinner <victor.stinner@haypocalc.com> parents: 68256 diff changeset	477 self.assertEqual(reader.read(), s1)
59169 4066b45d28ab Merged revisions 78461,78482 via svnmerge from Florent Xicluna <florent.xicluna@gmail.com> parents: 56034 diff changeset	478
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	479 class UTF16LETest(ReadTest):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	480 encoding = "utf-16-le"
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	481
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	482 def test_partial(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	483 self.check_partial(
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	484 u"\x00\xff\u0100\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	485 [
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	486 u"",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	487 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	488 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	489 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	490 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	491 u"\x00\xff\u0100",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	492 u"\x00\xff\u0100",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	493 u"\x00\xff\u0100\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	494 ]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	495 )
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	496
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	497 def test_errors(self):
78208 4cadf91aaddd Issue #14579: Fix error handling bug in the utf-16 decoder. Antoine Pitrou <solipsis@pitrou.net> parents: 71492 diff changeset	498 tests = [
4cadf91aaddd Issue #14579: Fix error handling bug in the utf-16 decoder. Antoine Pitrou <solipsis@pitrou.net> parents: 71492 diff changeset	499 (b'\xff', u'\ufffd'),
4cadf91aaddd Issue #14579: Fix error handling bug in the utf-16 decoder. Antoine Pitrou <solipsis@pitrou.net> parents: 71492 diff changeset	500 (b'A\x00Z', u'A\ufffd'),
4cadf91aaddd Issue #14579: Fix error handling bug in the utf-16 decoder. Antoine Pitrou <solipsis@pitrou.net> parents: 71492 diff changeset	501 (b'A\x00B\x00C\x00D\x00Z', u'ABCD\ufffd'),
4cadf91aaddd Issue #14579: Fix error handling bug in the utf-16 decoder. Antoine Pitrou <solipsis@pitrou.net> parents: 71492 diff changeset	502 (b'\x00\xd8', u'\ufffd'),
4cadf91aaddd Issue #14579: Fix error handling bug in the utf-16 decoder. Antoine Pitrou <solipsis@pitrou.net> parents: 71492 diff changeset	503 (b'\x00\xd8A', u'\ufffd'),
4cadf91aaddd Issue #14579: Fix error handling bug in the utf-16 decoder. Antoine Pitrou <solipsis@pitrou.net> parents: 71492 diff changeset	504 (b'\x00\xd8A\x00', u'\ufffdA'),
4cadf91aaddd Issue #14579: Fix error handling bug in the utf-16 decoder. Antoine Pitrou <solipsis@pitrou.net> parents: 71492 diff changeset	505 (b'\x00\xdcA\x00', u'\ufffdA'),
4cadf91aaddd Issue #14579: Fix error handling bug in the utf-16 decoder. Antoine Pitrou <solipsis@pitrou.net> parents: 71492 diff changeset	506 ]
4cadf91aaddd Issue #14579: Fix error handling bug in the utf-16 decoder. Antoine Pitrou <solipsis@pitrou.net> parents: 71492 diff changeset	507 for raw, expected in tests:
4cadf91aaddd Issue #14579: Fix error handling bug in the utf-16 decoder. Antoine Pitrou <solipsis@pitrou.net> parents: 71492 diff changeset	508 self.assertRaises(UnicodeDecodeError, codecs.utf_16_le_decode,
4cadf91aaddd Issue #14579: Fix error handling bug in the utf-16 decoder. Antoine Pitrou <solipsis@pitrou.net> parents: 71492 diff changeset	509 raw, 'strict', True)
4cadf91aaddd Issue #14579: Fix error handling bug in the utf-16 decoder. Antoine Pitrou <solipsis@pitrou.net> parents: 71492 diff changeset	510 self.assertEqual(raw.decode('utf-16le', 'replace'), expected)
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	511
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	512 class UTF16BETest(ReadTest):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	513 encoding = "utf-16-be"
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	514
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	515 def test_partial(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	516 self.check_partial(
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	517 u"\x00\xff\u0100\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	518 [
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	519 u"",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	520 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	521 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	522 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	523 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	524 u"\x00\xff\u0100",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	525 u"\x00\xff\u0100",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	526 u"\x00\xff\u0100\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	527 ]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	528 )
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	529
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	530 def test_errors(self):
78208 4cadf91aaddd Issue #14579: Fix error handling bug in the utf-16 decoder. Antoine Pitrou <solipsis@pitrou.net> parents: 71492 diff changeset	531 tests = [
4cadf91aaddd Issue #14579: Fix error handling bug in the utf-16 decoder. Antoine Pitrou <solipsis@pitrou.net> parents: 71492 diff changeset	532 (b'\xff', u'\ufffd'),
4cadf91aaddd Issue #14579: Fix error handling bug in the utf-16 decoder. Antoine Pitrou <solipsis@pitrou.net> parents: 71492 diff changeset	533 (b'\x00A\xff', u'A\ufffd'),
4cadf91aaddd Issue #14579: Fix error handling bug in the utf-16 decoder. Antoine Pitrou <solipsis@pitrou.net> parents: 71492 diff changeset	534 (b'\x00A\x00B\x00C\x00DZ', u'ABCD\ufffd'),
4cadf91aaddd Issue #14579: Fix error handling bug in the utf-16 decoder. Antoine Pitrou <solipsis@pitrou.net> parents: 71492 diff changeset	535 (b'\xd8\x00', u'\ufffd'),
4cadf91aaddd Issue #14579: Fix error handling bug in the utf-16 decoder. Antoine Pitrou <solipsis@pitrou.net> parents: 71492 diff changeset	536 (b'\xd8\x00\xdc', u'\ufffd'),
4cadf91aaddd Issue #14579: Fix error handling bug in the utf-16 decoder. Antoine Pitrou <solipsis@pitrou.net> parents: 71492 diff changeset	537 (b'\xd8\x00\x00A', u'\ufffdA'),
4cadf91aaddd Issue #14579: Fix error handling bug in the utf-16 decoder. Antoine Pitrou <solipsis@pitrou.net> parents: 71492 diff changeset	538 (b'\xdc\x00\x00A', u'\ufffdA'),
4cadf91aaddd Issue #14579: Fix error handling bug in the utf-16 decoder. Antoine Pitrou <solipsis@pitrou.net> parents: 71492 diff changeset	539 ]
4cadf91aaddd Issue #14579: Fix error handling bug in the utf-16 decoder. Antoine Pitrou <solipsis@pitrou.net> parents: 71492 diff changeset	540 for raw, expected in tests:
4cadf91aaddd Issue #14579: Fix error handling bug in the utf-16 decoder. Antoine Pitrou <solipsis@pitrou.net> parents: 71492 diff changeset	541 self.assertRaises(UnicodeDecodeError, codecs.utf_16_be_decode,
4cadf91aaddd Issue #14579: Fix error handling bug in the utf-16 decoder. Antoine Pitrou <solipsis@pitrou.net> parents: 71492 diff changeset	542 raw, 'strict', True)
4cadf91aaddd Issue #14579: Fix error handling bug in the utf-16 decoder. Antoine Pitrou <solipsis@pitrou.net> parents: 71492 diff changeset	543 self.assertEqual(raw.decode('utf-16be', 'replace'), expected)
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	544
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	545 class UTF8Test(ReadTest):
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33855 diff changeset	546 encoding = "utf-8"
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	547
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	548 def test_partial(self):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	549 self.check_partial(
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	550 u"\x00\xff\u07ff\u0800\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	551 [
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	552 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	553 u"\x00",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	554 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	555 u"\x00\xff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	556 u"\x00\xff\u07ff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	557 u"\x00\xff\u07ff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	558 u"\x00\xff\u07ff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	559 u"\x00\xff\u07ff\u0800",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	560 u"\x00\xff\u07ff\u0800",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	561 u"\x00\xff\u07ff\u0800",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	562 u"\x00\xff\u07ff\u0800\uffff",
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	563 ]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	564 )
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	565
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	566 class UTF7Test(ReadTest):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	567 encoding = "utf-7"
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	568
43932 e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	569 def test_partial(self):
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	570 self.check_partial(
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	571 u"a+-b",
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	572 [
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	573 u"a",
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	574 u"a",
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	575 u"a+",
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	576 u"a+-",
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	577 u"a+-b",
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	578 ]
e8ac1ae4503c The incremental decoder for utf-7 must preserve its state between calls. Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 43905 diff changeset	579 )
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	580
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	581 class UTF16ExTest(unittest.TestCase):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	582
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	583 def test_errors(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	584 self.assertRaises(UnicodeDecodeError, codecs.utf_16_ex_decode, "\xff", "strict", 0, True)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	585
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	586 def test_bad_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	587 self.assertRaises(TypeError, codecs.utf_16_ex_decode)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	588
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	589 class ReadBufferTest(unittest.TestCase):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	590
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	591 def test_array(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	592 import array
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	593 self.assertEqual(
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	594 codecs.readbuffer_encode(array.array("c", "spam")),
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	595 ("spam", 4)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	596 )
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	597
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	598 def test_empty(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	599 self.assertEqual(codecs.readbuffer_encode(""), ("", 0))
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	600
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	601 def test_bad_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	602 self.assertRaises(TypeError, codecs.readbuffer_encode)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	603 self.assertRaises(TypeError, codecs.readbuffer_encode, 42)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	604
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	605 class CharBufferTest(unittest.TestCase):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	606
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	607 def test_string(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	608 self.assertEqual(codecs.charbuffer_encode("spam"), ("spam", 4))
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	609
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	610 def test_empty(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	611 self.assertEqual(codecs.charbuffer_encode(""), ("", 0))
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	612
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	613 def test_bad_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	614 self.assertRaises(TypeError, codecs.charbuffer_encode)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	615 self.assertRaises(TypeError, codecs.charbuffer_encode, 42)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	616
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	617 class UTF8SigTest(ReadTest):
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	618 encoding = "utf-8-sig"
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	619
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	620 def test_partial(self):
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	621 self.check_partial(
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	622 u"\ufeff\x00\xff\u07ff\u0800\uffff",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	623 [
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	624 u"",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	625 u"",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	626 u"", # First BOM has been read and skipped
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	627 u"",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	628 u"",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	629 u"\ufeff", # Second BOM has been read and emitted
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	630 u"\ufeff\x00", # "\x00" read and emitted
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	631 u"\ufeff\x00", # First byte of encoded u"\xff" read
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	632 u"\ufeff\x00\xff", # Second byte of encoded u"\xff" read
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	633 u"\ufeff\x00\xff", # First byte of encoded u"\u07ff" read
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	634 u"\ufeff\x00\xff\u07ff", # Second byte of encoded u"\u07ff" read
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	635 u"\ufeff\x00\xff\u07ff",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	636 u"\ufeff\x00\xff\u07ff",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	637 u"\ufeff\x00\xff\u07ff\u0800",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	638 u"\ufeff\x00\xff\u07ff\u0800",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	639 u"\ufeff\x00\xff\u07ff\u0800",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	640 u"\ufeff\x00\xff\u07ff\u0800\uffff",
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	641 ]
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	642 )
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	643
40387 b6b025931451 Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 40253 diff changeset	644 def test_bug1601501(self):
b6b025931451 Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 40253 diff changeset	645 # SF bug #1601501: check that the codec works with a buffer
b6b025931451 Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 40253 diff changeset	646 unicode("\xef\xbb\xbf", "utf-8-sig")
b6b025931451 Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 40253 diff changeset	647
41498 5a4619214043 Backport r54786: Walter Dörwald <walter@livinglogic.de> parents: 40387 diff changeset	648 def test_bom(self):
5a4619214043 Backport r54786: Walter Dörwald <walter@livinglogic.de> parents: 40387 diff changeset	649 d = codecs.getincrementaldecoder("utf-8-sig")()
5a4619214043 Backport r54786: Walter Dörwald <walter@livinglogic.de> parents: 40387 diff changeset	650 s = u"spam"
5a4619214043 Backport r54786: Walter Dörwald <walter@livinglogic.de> parents: 40387 diff changeset	651 self.assertEqual(d.decode(s.encode("utf-8-sig")), s)
5a4619214043 Backport r54786: Walter Dörwald <walter@livinglogic.de> parents: 40387 diff changeset	652
43906 e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	653 def test_stream_bom(self):
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	654 unistring = u"ABC\u00A1\u2200XYZ"
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	655 bytestring = codecs.BOM_UTF8 + "ABC\xC2\xA1\xE2\x88\x80XYZ"
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	656
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	657 reader = codecs.getreader("utf-8-sig")
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	658 for sizehint in [None] + range(1, 11) + \
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	659 [64, 128, 256, 512, 1024]:
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	660 istream = reader(StringIO.StringIO(bytestring))
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	661 ostream = StringIO.StringIO()
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	662 while 1:
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	663 if sizehint is not None:
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	664 data = istream.read(sizehint)
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	665 else:
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	666 data = istream.read()
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	667
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	668 if not data:
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	669 break
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	670 ostream.write(data)
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	671
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	672 got = ostream.getvalue()
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	673 self.assertEqual(got, unistring)
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	674
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	675 def test_stream_bare(self):
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	676 unistring = u"ABC\u00A1\u2200XYZ"
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	677 bytestring = "ABC\xC2\xA1\xE2\x88\x80XYZ"
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	678
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	679 reader = codecs.getreader("utf-8-sig")
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	680 for sizehint in [None] + range(1, 11) + \
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	681 [64, 128, 256, 512, 1024]:
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	682 istream = reader(StringIO.StringIO(bytestring))
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	683 ostream = StringIO.StringIO()
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	684 while 1:
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	685 if sizehint is not None:
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	686 data = istream.read(sizehint)
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	687 else:
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	688 data = istream.read()
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	689
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	690 if not data:
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	691 break
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	692 ostream.write(data)
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	693
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	694 got = ostream.getvalue()
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	695 self.assertEqual(got, unistring)
e5d48fbe1646 Backport r59049: Walter Dörwald <walter@livinglogic.de> parents: 43904 diff changeset	696
25100 bc590beadf0b Check whether a string resize is necessary at the end Walter Dörwald <walter@livinglogic.de> parents: 24364 diff changeset	697 class EscapeDecodeTest(unittest.TestCase):
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	698 def test_empty(self):
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	699 self.assertEqual(codecs.escape_decode(""), ("", 0))
25100 bc590beadf0b Check whether a string resize is necessary at the end Walter Dörwald <walter@livinglogic.de> parents: 24364 diff changeset	700
27680 b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	701 class RecodingTest(unittest.TestCase):
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	702 def test_recoding(self):
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	703 f = StringIO.StringIO()
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	704 f2 = codecs.EncodedFile(f, "unicode_internal", "utf-8")
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	705 f2.write(u"a")
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	706 f2.close()
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	707 # Python used to crash on this at exit because of a refcount
b19aaaeb6d64 Fix for [ 543344 ] Interpreter crashes when recoding; suggested Marc-André Lemburg <mal@egenix.com> parents: 25100 diff changeset	708 # bug in _codecsmodule.c
20013 e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	709
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	710 # From RFC 3492
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	711 punycode_testcases = [
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	712 # A Arabic (Egyptian):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	713 (u"\u0644\u064A\u0647\u0645\u0627\u0628\u062A\u0643\u0644"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	714 u"\u0645\u0648\u0634\u0639\u0631\u0628\u064A\u061F",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	715 "egbpdaj6bu4bxfgehfvwxn"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	716 # B Chinese (simplified):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	717 (u"\u4ED6\u4EEC\u4E3A\u4EC0\u4E48\u4E0D\u8BF4\u4E2D\u6587",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	718 "ihqwcrb4cv8a8dqg056pqjye"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	719 # C Chinese (traditional):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	720 (u"\u4ED6\u5011\u7232\u4EC0\u9EBD\u4E0D\u8AAA\u4E2D\u6587",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	721 "ihqwctvzc91f659drss3x8bo0yb"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	722 # D Czech: Pro<ccaron>prost<ecaron>nemluv<iacute><ccaron>esky
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	723 (u"\u0050\u0072\u006F\u010D\u0070\u0072\u006F\u0073\u0074"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	724 u"\u011B\u006E\u0065\u006D\u006C\u0075\u0076\u00ED\u010D"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	725 u"\u0065\u0073\u006B\u0079",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	726 "Proprostnemluvesky-uyb24dma41a"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	727 # E Hebrew:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	728 (u"\u05DC\u05DE\u05D4\u05D4\u05DD\u05E4\u05E9\u05D5\u05D8"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	729 u"\u05DC\u05D0\u05DE\u05D3\u05D1\u05E8\u05D9\u05DD\u05E2"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	730 u"\u05D1\u05E8\u05D9\u05EA",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	731 "4dbcagdahymbxekheh6e0a7fei0b"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	732 # F Hindi (Devanagari):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	733 (u"\u092F\u0939\u0932\u094B\u0917\u0939\u093F\u0928\u094D"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	734 u"\u0926\u0940\u0915\u094D\u092F\u094B\u0902\u0928\u0939"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	735 u"\u0940\u0902\u092C\u094B\u0932\u0938\u0915\u0924\u0947"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	736 u"\u0939\u0948\u0902",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	737 "i1baa7eci9glrd9b2ae1bj0hfcgg6iyaf8o0a1dig0cd"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	738
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	739 #(G) Japanese (kanji and hiragana):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	740 (u"\u306A\u305C\u307F\u3093\u306A\u65E5\u672C\u8A9E\u3092"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	741 u"\u8A71\u3057\u3066\u304F\u308C\u306A\u3044\u306E\u304B",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	742 "n8jok5ay5dzabd5bym9f0cm5685rrjetr6pdxa"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	743
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	744 # (H) Korean (Hangul syllables):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	745 (u"\uC138\uACC4\uC758\uBAA8\uB4E0\uC0AC\uB78C\uB4E4\uC774"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	746 u"\uD55C\uAD6D\uC5B4\uB97C\uC774\uD574\uD55C\uB2E4\uBA74"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	747 u"\uC5BC\uB9C8\uB098\uC88B\uC744\uAE4C",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	748 "989aomsvi5e83db1d2a355cv1e0vak1dwrv93d5xbh15a0dt30a5j"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	749 "psd879ccm6fea98c"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	750
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	751 # (I) Russian (Cyrillic):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	752 (u"\u043F\u043E\u0447\u0435\u043C\u0443\u0436\u0435\u043E"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	753 u"\u043D\u0438\u043D\u0435\u0433\u043E\u0432\u043E\u0440"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	754 u"\u044F\u0442\u043F\u043E\u0440\u0443\u0441\u0441\u043A"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	755 u"\u0438",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	756 "b1abfaaepdrnnbgefbaDotcwatmq2g4l"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	757
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	758 # (J) Spanish: Porqu<eacute>nopuedensimplementehablarenEspa<ntilde>ol
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	759 (u"\u0050\u006F\u0072\u0071\u0075\u00E9\u006E\u006F\u0070"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	760 u"\u0075\u0065\u0064\u0065\u006E\u0073\u0069\u006D\u0070"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	761 u"\u006C\u0065\u006D\u0065\u006E\u0074\u0065\u0068\u0061"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	762 u"\u0062\u006C\u0061\u0072\u0065\u006E\u0045\u0073\u0070"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	763 u"\u0061\u00F1\u006F\u006C",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	764 "PorqunopuedensimplementehablarenEspaol-fmd56a"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	765
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	766 # (K) Vietnamese:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	767 # T<adotbelow>isaoh<odotbelow>kh<ocirc>ngth<ecirchookabove>ch\
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	768 # <ihookabove>n<oacute>iti<ecircacute>ngVi<ecircdotbelow>t
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	769 (u"\u0054\u1EA1\u0069\u0073\u0061\u006F\u0068\u1ECD\u006B"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	770 u"\u0068\u00F4\u006E\u0067\u0074\u0068\u1EC3\u0063\u0068"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	771 u"\u1EC9\u006E\u00F3\u0069\u0074\u0069\u1EBF\u006E\u0067"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	772 u"\u0056\u0069\u1EC7\u0074",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	773 "TisaohkhngthchnitingVit-kjcr8268qyxafd2f1b9g"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	774
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	775 #(L) 3<nen>B<gumi><kinpachi><sensei>
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	776 (u"\u0033\u5E74\u0042\u7D44\u91D1\u516B\u5148\u751F",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	777 "3B-ww4c5e180e575a65lsy2b"),
28774 c9bee11eaa8b Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 28678 diff changeset	778
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	779 # (M) <amuro><namie>-with-SUPER-MONKEYS
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	780 (u"\u5B89\u5BA4\u5948\u7F8E\u6075\u002D\u0077\u0069\u0074"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	781 u"\u0068\u002D\u0053\u0055\u0050\u0045\u0052\u002D\u004D"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	782 u"\u004F\u004E\u004B\u0045\u0059\u0053",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	783 "-with-SUPER-MONKEYS-pc58ag80a8qai00g7n9n"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	784
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	785 # (N) Hello-Another-Way-<sorezore><no><basho>
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	786 (u"\u0048\u0065\u006C\u006C\u006F\u002D\u0041\u006E\u006F"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	787 u"\u0074\u0068\u0065\u0072\u002D\u0057\u0061\u0079\u002D"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	788 u"\u305D\u308C\u305E\u308C\u306E\u5834\u6240",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	789 "Hello-Another-Way--fc4qua05auwb3674vfr0b"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	790
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	791 # (O) <hitotsu><yane><no><shita>2
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	792 (u"\u3072\u3068\u3064\u5C4B\u6839\u306E\u4E0B\u0032",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	793 "2-u9tlzr9756bt3uc0v"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	794
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	795 # (P) Maji<de>Koi<suru>5<byou><mae>
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	796 (u"\u004D\u0061\u006A\u0069\u3067\u004B\u006F\u0069\u3059"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	797 u"\u308B\u0035\u79D2\u524D",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	798 "MajiKoi5-783gue6qz075azm5e"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	799
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	800 # (Q) <pafii>de<runba>
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	801 (u"\u30D1\u30D5\u30A3\u30FC\u0064\u0065\u30EB\u30F3\u30D0",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	802 "de-jg4avhby1noc0d"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	803
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	804 # (R) <sono><supiido><de>
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	805 (u"\u305D\u306E\u30B9\u30D4\u30FC\u30C9\u3067",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	806 "d9juau41awczczp"),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	807
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	808 # (S) -> $1.00 <-
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	809 (u"\u002D\u003E\u0020\u0024\u0031\u002E\u0030\u0030\u0020"
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	810 u"\u003C\u002D",
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	811 "-> $1.00 <--")
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	812 ]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	813
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	814 for i in punycode_testcases:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	815 if len(i)!=2:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	816 print repr(i)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	817
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	818 class PunycodeTest(unittest.TestCase):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	819 def test_encode(self):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	820 for uni, puny in punycode_testcases:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	821 # Need to convert both strings to lower case, since
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	822 # some of the extended encodings use upper case, but our
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	823 # code produces only lower case. Converting just puny to
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	824 # lower is also insufficient, since some of the input characters
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	825 # are upper case.
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	826 self.assertEqual(uni.encode("punycode").lower(), puny.lower())
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	827
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	828 def test_decode(self):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	829 for uni, puny in punycode_testcases:
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	830 self.assertEqual(uni, puny.decode("punycode"))
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	831
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	832 class UnicodeInternalTest(unittest.TestCase):
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	833 def test_bug1251300(self):
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	834 # Decoding with unicode_internal used to not correctly handle "code
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	835 # points" above 0x10ffff on UCS-4 builds.
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	836 if sys.maxunicode > 0xffff:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	837 ok = [
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	838 ("\x00\x10\xff\xff", u"\U0010ffff"),
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	839 ("\x00\x00\x01\x01", u"\U00000101"),
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	840 ("", u""),
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	841 ]
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	842 not_ok = [
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	843 "\x7f\xff\xff\xff",
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	844 "\x80\x00\x00\x00",
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	845 "\x81\x00\x00\x00",
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	846 "\x00",
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	847 "\x00\x00\x00\x00\x00",
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	848 ]
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	849 for internal, uni in ok:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	850 if sys.byteorder == "little":
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	851 internal = "".join(reversed(internal))
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	852 self.assertEqual(uni, internal.decode("unicode_internal"))
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	853 for internal in not_ok:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	854 if sys.byteorder == "little":
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	855 internal = "".join(reversed(internal))
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	856 self.assertRaises(UnicodeDecodeError, internal.decode,
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	857 "unicode_internal")
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	858
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	859 def test_decode_error_attributes(self):
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	860 if sys.maxunicode > 0xffff:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	861 try:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	862 "\x00\x00\x00\x00\x00\x11\x11\x00".decode("unicode_internal")
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	863 except UnicodeDecodeError, ex:
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	864 self.assertEqual("unicode_internal", ex.encoding)
614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	865 self.assertEqual("\x00\x00\x00\x00\x00\x11\x11\x00", ex.object)
614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	866 self.assertEqual(4, ex.start)
614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	867 self.assertEqual(8, ex.end)
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	868 else:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	869 self.fail()
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	870
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	871 def test_decode_callback(self):
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	872 if sys.maxunicode > 0xffff:
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	873 codecs.register_error("UnicodeInternalTest", codecs.ignore_errors)
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	874 decoder = codecs.getdecoder("unicode_internal")
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	875 ab = u"ab".encode("unicode_internal")
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	876 ignored = decoder("%s\x22\x22\x22\x22%s" % (ab[:4], ab[4:]),
4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	877 "UnicodeInternalTest")
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	878 self.assertEqual((u"ab", 12), ignored)
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	879
53855 7dca250343ea Issue 3739: The unicode-internal encoder now reports the number of characters Walter Dörwald <walter@livinglogic.de> parents: 43932 diff changeset	880 def test_encode_length(self):
7dca250343ea Issue 3739: The unicode-internal encoder now reports the number of characters Walter Dörwald <walter@livinglogic.de> parents: 43932 diff changeset	881 # Issue 3739
7dca250343ea Issue 3739: The unicode-internal encoder now reports the number of characters Walter Dörwald <walter@livinglogic.de> parents: 43932 diff changeset	882 encoder = codecs.getencoder("unicode_internal")
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	883 self.assertEqual(encoder(u"a")[1], 1)
614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	884 self.assertEqual(encoder(u"\xe9\u0142")[1], 2)
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	885
62214 a2ea1a23186b Merged revisions 79779 via svnmerge from Philip Jenvey <pjenvey@underboss.org> parents: 61861 diff changeset	886 encoder = codecs.getencoder("string-escape")
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	887 self.assertEqual(encoder(r'\x00')[1], 4)
62214 a2ea1a23186b Merged revisions 79779 via svnmerge from Philip Jenvey <pjenvey@underboss.org> parents: 61861 diff changeset	888
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	889 # From http://www.gnu.org/software/libidn/draft-josefsson-idn-test-vectors.html
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	890 nameprep_tests = [
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	891 # 3.1 Map to nothing.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	892 ('foo\xc2\xad\xcd\x8f\xe1\xa0\x86\xe1\xa0\x8bbar'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	893 '\xe2\x80\x8b\xe2\x81\xa0baz\xef\xb8\x80\xef\xb8\x88\xef'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	894 '\xb8\x8f\xef\xbb\xbf',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	895 'foobarbaz'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	896 # 3.2 Case folding ASCII U+0043 U+0041 U+0046 U+0045.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	897 ('CAFE',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	898 'cafe'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	899 # 3.3 Case folding 8bit U+00DF (german sharp s).
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	900 # The original test case is bogus; it says \xc3\xdf
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	901 ('\xc3\x9f',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	902 'ss'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	903 # 3.4 Case folding U+0130 (turkish capital I with dot).
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	904 ('\xc4\xb0',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	905 'i\xcc\x87'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	906 # 3.5 Case folding multibyte U+0143 U+037A.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	907 ('\xc5\x83\xcd\xba',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	908 '\xc5\x84 \xce\xb9'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	909 # 3.6 Case folding U+2121 U+33C6 U+1D7BB.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	910 # XXX: skip this as it fails in UCS-2 mode
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	911 #('\xe2\x84\xa1\xe3\x8f\x86\xf0\x9d\x9e\xbb',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	912 # 'telc\xe2\x88\x95kg\xcf\x83'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	913 (None, None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	914 # 3.7 Normalization of U+006a U+030c U+00A0 U+00AA.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	915 ('j\xcc\x8c\xc2\xa0\xc2\xaa',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	916 '\xc7\xb0 a'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	917 # 3.8 Case folding U+1FB7 and normalization.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	918 ('\xe1\xbe\xb7',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	919 '\xe1\xbe\xb6\xce\xb9'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	920 # 3.9 Self-reverting case folding U+01F0 and normalization.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	921 # The original test case is bogus, it says `\xc7\xf0'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	922 ('\xc7\xb0',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	923 '\xc7\xb0'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	924 # 3.10 Self-reverting case folding U+0390 and normalization.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	925 ('\xce\x90',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	926 '\xce\x90'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	927 # 3.11 Self-reverting case folding U+03B0 and normalization.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	928 ('\xce\xb0',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	929 '\xce\xb0'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	930 # 3.12 Self-reverting case folding U+1E96 and normalization.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	931 ('\xe1\xba\x96',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	932 '\xe1\xba\x96'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	933 # 3.13 Self-reverting case folding U+1F56 and normalization.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	934 ('\xe1\xbd\x96',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	935 '\xe1\xbd\x96'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	936 # 3.14 ASCII space character U+0020.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	937 (' ',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	938 ' '),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	939 # 3.15 Non-ASCII 8bit space character U+00A0.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	940 ('\xc2\xa0',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	941 ' '),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	942 # 3.16 Non-ASCII multibyte space character U+1680.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	943 ('\xe1\x9a\x80',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	944 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	945 # 3.17 Non-ASCII multibyte space character U+2000.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	946 ('\xe2\x80\x80',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	947 ' '),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	948 # 3.18 Zero Width Space U+200b.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	949 ('\xe2\x80\x8b',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	950 ''),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	951 # 3.19 Non-ASCII multibyte space character U+3000.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	952 ('\xe3\x80\x80',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	953 ' '),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	954 # 3.20 ASCII control characters U+0010 U+007F.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	955 ('\x10\x7f',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	956 '\x10\x7f'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	957 # 3.21 Non-ASCII 8bit control character U+0085.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	958 ('\xc2\x85',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	959 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	960 # 3.22 Non-ASCII multibyte control character U+180E.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	961 ('\xe1\xa0\x8e',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	962 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	963 # 3.23 Zero Width No-Break Space U+FEFF.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	964 ('\xef\xbb\xbf',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	965 ''),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	966 # 3.24 Non-ASCII control character U+1D175.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	967 ('\xf0\x9d\x85\xb5',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	968 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	969 # 3.25 Plane 0 private use character U+F123.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	970 ('\xef\x84\xa3',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	971 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	972 # 3.26 Plane 15 private use character U+F1234.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	973 ('\xf3\xb1\x88\xb4',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	974 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	975 # 3.27 Plane 16 private use character U+10F234.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	976 ('\xf4\x8f\x88\xb4',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	977 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	978 # 3.28 Non-character code point U+8FFFE.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	979 ('\xf2\x8f\xbf\xbe',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	980 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	981 # 3.29 Non-character code point U+10FFFF.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	982 ('\xf4\x8f\xbf\xbf',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	983 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	984 # 3.30 Surrogate code U+DF42.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	985 ('\xed\xbd\x82',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	986 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	987 # 3.31 Non-plain text character U+FFFD.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	988 ('\xef\xbf\xbd',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	989 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	990 # 3.32 Ideographic description character U+2FF5.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	991 ('\xe2\xbf\xb5',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	992 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	993 # 3.33 Display property character U+0341.
28774 c9bee11eaa8b Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 28678 diff changeset	994 ('\xcd\x81',
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	995 '\xcc\x81'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	996 # 3.34 Left-to-right mark U+200E.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	997 ('\xe2\x80\x8e',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	998 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	999 # 3.35 Deprecated U+202A.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1000 ('\xe2\x80\xaa',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1001 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1002 # 3.36 Language tagging character U+E0001.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1003 ('\xf3\xa0\x80\x81',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1004 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1005 # 3.37 Language tagging character U+E0042.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1006 ('\xf3\xa0\x81\x82',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1007 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1008 # 3.38 Bidi: RandALCat character U+05BE and LCat characters.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1009 ('foo\xd6\xbebar',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1010 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1011 # 3.39 Bidi: RandALCat character U+FD50 and LCat characters.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1012 ('foo\xef\xb5\x90bar',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1013 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1014 # 3.40 Bidi: RandALCat character U+FB38 and LCat characters.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1015 ('foo\xef\xb9\xb6bar',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1016 'foo \xd9\x8ebar'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1017 # 3.41 Bidi: RandALCat without trailing RandALCat U+0627 U+0031.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1018 ('\xd8\xa71',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1019 None),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1020 # 3.42 Bidi: RandALCat character U+0627 U+0031 U+0628.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1021 ('\xd8\xa71\xd8\xa8',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1022 '\xd8\xa71\xd8\xa8'),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1023 # 3.43 Unassigned code point U+E0002.
28678 31cb8e0260db Skip nameprep test 3.43, as we do allow unassigned characters. The test Martin v. Löwis <martin@v.loewis.de> parents: 28671 diff changeset	1024 # Skip this test as we allow unassigned
31cb8e0260db Skip nameprep test 3.43, as we do allow unassigned characters. The test Martin v. Löwis <martin@v.loewis.de> parents: 28671 diff changeset	1025 #('\xf3\xa0\x80\x82',
31cb8e0260db Skip nameprep test 3.43, as we do allow unassigned characters. The test Martin v. Löwis <martin@v.loewis.de> parents: 28671 diff changeset	1026 # None),
31cb8e0260db Skip nameprep test 3.43, as we do allow unassigned characters. The test Martin v. Löwis <martin@v.loewis.de> parents: 28671 diff changeset	1027 (None, None),
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1028 # 3.44 Larger test (shrinking).
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1029 # Original test case reads \xc3\xdf
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1030 ('X\xc2\xad\xc3\x9f\xc4\xb0\xe2\x84\xa1j\xcc\x8c\xc2\xa0\xc2'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1031 '\xaa\xce\xb0\xe2\x80\x80',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1032 'xssi\xcc\x87tel\xc7\xb0 a\xce\xb0 '),
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1033 # 3.45 Larger test (expanding).
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1034 # Original test case reads \xc3\x9f
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1035 ('X\xc3\x9f\xe3\x8c\x96\xc4\xb0\xe2\x84\xa1\xe2\x92\x9f\xe3\x8c'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1036 '\x80',
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1037 'xss\xe3\x82\xad\xe3\x83\xad\xe3\x83\xa1\xe3\x83\xbc\xe3'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1038 '\x83\x88\xe3\x83\xabi\xcc\x87tel\x28d\x29\xe3\x82'
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1039 '\xa2\xe3\x83\x91\xe3\x83\xbc\xe3\x83\x88')
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1040 ]
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1041
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1042
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1043 class NameprepTest(unittest.TestCase):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1044 def test_nameprep(self):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1045 from encodings.idna import nameprep
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1046 for pos, (orig, prepped) in enumerate(nameprep_tests):
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1047 if orig is None:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1048 # Skipped
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1049 continue
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1050 # The Unicode strings are given in UTF-8
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1051 orig = unicode(orig, "utf-8")
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1052 if prepped is None:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1053 # Input contains prohibited characters
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1054 self.assertRaises(UnicodeError, nameprep, orig)
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1055 else:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1056 prepped = unicode(prepped, "utf-8")
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1057 try:
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1058 self.assertEqual(nameprep(orig), prepped)
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1059 except Exception,e:
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1060 raise test_support.TestFailed("Test 3.%d: %s" % (pos+1, str(e)))
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27680 diff changeset	1061
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1062 class IDNACodecTest(unittest.TestCase):
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1063 def test_builtin_decode(self):
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1064 self.assertEqual(unicode("python.org", "idna"), u"python.org")
614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1065 self.assertEqual(unicode("python.org.", "idna"), u"python.org.")
614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1066 self.assertEqual(unicode("xn--pythn-mua.org", "idna"), u"pyth\xf6n.org")
614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1067 self.assertEqual(unicode("xn--pythn-mua.org.", "idna"), u"pyth\xf6n.org.")
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1068
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1069 def test_builtin_encode(self):
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1070 self.assertEqual(u"python.org".encode("idna"), "python.org")
614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1071 self.assertEqual("python.org.".encode("idna"), "python.org.")
614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1072 self.assertEqual(u"pyth\xf6n.org".encode("idna"), "xn--pythn-mua.org")
614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1073 self.assertEqual(u"pyth\xf6n.org.".encode("idna"), "xn--pythn-mua.org.")
31795 bf37429f6d9f Add test case for unicode(somestring, "idna"). Martin v. Löwis <martin@v.loewis.de> parents: 28917 diff changeset	1074
35435 9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	1075 def test_stream(self):
9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	1076 import StringIO
9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	1077 r = codecs.getreader("idna")(StringIO.StringIO("abc"))
9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	1078 r.read(3)
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1079 self.assertEqual(r.read(), u"")
35435 9d343bc81863 Make IDNA return an empty string when the input is empty. Fixes #1163178. Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	1080
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1081 def test_incremental_decode(self):
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1082 self.assertEqual(
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1083 "".join(codecs.iterdecode("python.org", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1084 u"python.org"
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1085 )
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1086 self.assertEqual(
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1087 "".join(codecs.iterdecode("python.org.", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1088 u"python.org."
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1089 )
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1090 self.assertEqual(
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1091 "".join(codecs.iterdecode("xn--pythn-mua.org.", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1092 u"pyth\xf6n.org."
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1093 )
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1094 self.assertEqual(
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1095 "".join(codecs.iterdecode("xn--pythn-mua.org.", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1096 u"pyth\xf6n.org."
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1097 )
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1098
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1099 decoder = codecs.getincrementaldecoder("idna")()
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1100 self.assertEqual(decoder.decode("xn--xam", ), u"")
614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1101 self.assertEqual(decoder.decode("ple-9ta.o", ), u"\xe4xample.")
614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1102 self.assertEqual(decoder.decode(u"rg"), u"")
614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1103 self.assertEqual(decoder.decode(u"", True), u"org")
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1104
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1105 decoder.reset()
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1106 self.assertEqual(decoder.decode("xn--xam", ), u"")
614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1107 self.assertEqual(decoder.decode("ple-9ta.o", ), u"\xe4xample.")
614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1108 self.assertEqual(decoder.decode("rg."), u"org.")
614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1109 self.assertEqual(decoder.decode("", True), u"")
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1110
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1111 def test_incremental_encode(self):
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1112 self.assertEqual(
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1113 "".join(codecs.iterencode(u"python.org", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1114 "python.org"
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1115 )
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1116 self.assertEqual(
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1117 "".join(codecs.iterencode(u"python.org.", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1118 "python.org."
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1119 )
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1120 self.assertEqual(
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1121 "".join(codecs.iterencode(u"pyth\xf6n.org.", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1122 "xn--pythn-mua.org."
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1123 )
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1124 self.assertEqual(
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1125 "".join(codecs.iterencode(u"pyth\xf6n.org.", "idna")),
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1126 "xn--pythn-mua.org."
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1127 )
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1128
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1129 encoder = codecs.getincrementalencoder("idna")()
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1130 self.assertEqual(encoder.encode(u"\xe4x"), "")
614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1131 self.assertEqual(encoder.encode(u"ample.org"), "xn--xample-9ta.")
614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1132 self.assertEqual(encoder.encode(u"", True), "org")
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1133
cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1134 encoder.reset()
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1135 self.assertEqual(encoder.encode(u"\xe4x"), "")
614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1136 self.assertEqual(encoder.encode(u"ample.org."), "xn--xample-9ta.org.")
614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1137 self.assertEqual(encoder.encode(u"", True), "")
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1138
32572 8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1139 class CodecsModuleTest(unittest.TestCase):
8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1140
8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1141 def test_decode(self):
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1142 self.assertEqual(codecs.decode('\xe4\xf6\xfc', 'latin-1'),
32572 8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1143 u'\xe4\xf6\xfc')
33855 1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1144 self.assertRaises(TypeError, codecs.decode)
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1145 self.assertEqual(codecs.decode('abc'), u'abc')
33855 1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1146 self.assertRaises(UnicodeDecodeError, codecs.decode, '\xff', 'ascii')
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1147
32572 8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1148 def test_encode(self):
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1149 self.assertEqual(codecs.encode(u'\xe4\xf6\xfc', 'latin-1'),
32572 8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1150 '\xe4\xf6\xfc')
33855 1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1151 self.assertRaises(TypeError, codecs.encode)
35889 e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1152 self.assertRaises(LookupError, codecs.encode, "foo", "__spam__")
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1153 self.assertEqual(codecs.encode(u'abc'), 'abc')
33855 1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1154 self.assertRaises(UnicodeEncodeError, codecs.encode, u'\xffff', 'ascii')
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1155
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1156 def test_register(self):
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1157 self.assertRaises(TypeError, codecs.register)
35889 e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1158 self.assertRaises(TypeError, codecs.register, 42)
33855 1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1159
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1160 def test_lookup(self):
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1161 self.assertRaises(TypeError, codecs.lookup)
1cd19daaa9d4 Trigger a few error cases in Modules/_codecsmodule.c. Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	1162 self.assertRaises(LookupError, codecs.lookup, "__spam__")
35889 e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1163 self.assertRaises(LookupError, codecs.lookup, " ")
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1164
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1165 def test_getencoder(self):
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1166 self.assertRaises(TypeError, codecs.getencoder)
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1167 self.assertRaises(LookupError, codecs.getencoder, "__spam__")
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1168
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1169 def test_getdecoder(self):
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1170 self.assertRaises(TypeError, codecs.getdecoder)
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1171 self.assertRaises(LookupError, codecs.getdecoder, "__spam__")
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1172
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1173 def test_getreader(self):
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1174 self.assertRaises(TypeError, codecs.getreader)
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1175 self.assertRaises(LookupError, codecs.getreader, "__spam__")
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1176
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1177 def test_getwriter(self):
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1178 self.assertRaises(TypeError, codecs.getwriter)
e2473c4f2ee0 Add tests to increase code coverage in Python/codecs.c and Python/exceptions.c. Walter Dörwald <walter@livinglogic.de> parents: 35882 diff changeset	1179 self.assertRaises(LookupError, codecs.getwriter, "__spam__")
32572 8ea2cb1ec598 Add generic codecs.encode() and .decode() APIs that don't impose Marc-André Lemburg <mal@egenix.com> parents: 32538 diff changeset	1180
71492 739958134fe5 Issue #1813: Fix codec lookup and setting/getting locales under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70295 diff changeset	1181 def test_lookup_issue1813(self):
739958134fe5 Issue #1813: Fix codec lookup and setting/getting locales under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70295 diff changeset	1182 # Issue #1813: under Turkish locales, lookup of some codecs failed
739958134fe5 Issue #1813: Fix codec lookup and setting/getting locales under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70295 diff changeset	1183 # because 'I' is lowercased as a dotless "i"
739958134fe5 Issue #1813: Fix codec lookup and setting/getting locales under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70295 diff changeset	1184 oldlocale = locale.getlocale(locale.LC_CTYPE)
739958134fe5 Issue #1813: Fix codec lookup and setting/getting locales under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70295 diff changeset	1185 self.addCleanup(locale.setlocale, locale.LC_CTYPE, oldlocale)
739958134fe5 Issue #1813: Fix codec lookup and setting/getting locales under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70295 diff changeset	1186 try:
739958134fe5 Issue #1813: Fix codec lookup and setting/getting locales under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70295 diff changeset	1187 locale.setlocale(locale.LC_CTYPE, 'tr_TR')
739958134fe5 Issue #1813: Fix codec lookup and setting/getting locales under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70295 diff changeset	1188 except locale.Error:
739958134fe5 Issue #1813: Fix codec lookup and setting/getting locales under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70295 diff changeset	1189 # Unsupported locale on this system
739958134fe5 Issue #1813: Fix codec lookup and setting/getting locales under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70295 diff changeset	1190 self.skipTest('test needs Turkish locale')
739958134fe5 Issue #1813: Fix codec lookup and setting/getting locales under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70295 diff changeset	1191 c = codecs.lookup('ASCII')
739958134fe5 Issue #1813: Fix codec lookup and setting/getting locales under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70295 diff changeset	1192 self.assertEqual(c.name, 'ascii')
739958134fe5 Issue #1813: Fix codec lookup and setting/getting locales under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 70295 diff changeset	1193
33782 a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1194 class StreamReaderTest(unittest.TestCase):
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1195
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1196 def setUp(self):
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1197 self.reader = codecs.getreader('utf-8')
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1198 self.stream = StringIO.StringIO('\xed\x95\x9c\n\xea\xb8\x80')
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1199
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1200 def test_readlines(self):
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1201 f = self.reader(self.stream)
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1202 self.assertEqual(f.readlines(), [u'\ud55c\n', u'\uae00'])
33782 a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1203
40240 d245efc7ec89 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1204 class EncodedFileTest(unittest.TestCase):
41533 7ded787e6311 Whitespace normalization Neal Norwitz <nnorwitz@gmail.com> parents: 41498 diff changeset	1205
40240 d245efc7ec89 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1206 def test_basic(self):
d245efc7ec89 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1207 f = StringIO.StringIO('\xed\x95\x9c\n\xea\xb8\x80')
40250 7c44144714f6 Fix the new EncodedFile test to work with big endian platforms. Georg Brandl <georg@python.org> parents: 40240 diff changeset	1208 ef = codecs.EncodedFile(f, 'utf-16-le', 'utf-8')
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1209 self.assertEqual(ef.read(), '\\\xd5\n\x00\x00\xae')
40240 d245efc7ec89 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1210
d245efc7ec89 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1211 f = StringIO.StringIO()
d245efc7ec89 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1212 ef = codecs.EncodedFile(f, 'utf-8', 'latin1')
d245efc7ec89 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1213 ef.write('\xc3\xbc')
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1214 self.assertEqual(f.getvalue(), '\xfc')
40240 d245efc7ec89 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1215
35269 2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1216 class Str2StrTest(unittest.TestCase):
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1217
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1218 def test_read(self):
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1219 sin = "\x80".encode("base64_codec")
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1220 reader = codecs.getreader("base64_codec")(StringIO.StringIO(sin))
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1221 sout = reader.read()
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1222 self.assertEqual(sout, "\x80")
58472 6ffcdafa0118 use assert[Not]IsInstance where appropriate Ezio Melotti <ezio.melotti@gmail.com> parents: 56033 diff changeset	1223 self.assertIsInstance(sout, str)
35269 2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1224
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1225 def test_readline(self):
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1226 sin = "\x80".encode("base64_codec")
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1227 reader = codecs.getreader("base64_codec")(StringIO.StringIO(sin))
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1228 sout = reader.readline()
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1229 self.assertEqual(sout, "\x80")
58472 6ffcdafa0118 use assert[Not]IsInstance where appropriate Ezio Melotti <ezio.melotti@gmail.com> parents: 56033 diff changeset	1230 self.assertIsInstance(sout, str)
35269 2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1231
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1232 all_unicode_encodings = [
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1233 "ascii",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1234 "base64_codec",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1235 "big5",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1236 "big5hkscs",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1237 "charmap",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1238 "cp037",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1239 "cp1006",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1240 "cp1026",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1241 "cp1140",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1242 "cp1250",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1243 "cp1251",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1244 "cp1252",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1245 "cp1253",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1246 "cp1254",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1247 "cp1255",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1248 "cp1256",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1249 "cp1257",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1250 "cp1258",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1251 "cp424",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1252 "cp437",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1253 "cp500",
61884 c56fb63c648f #8016: add the CP858 codec (approved by Benjamin). (Also add CP720 to the tests, it was missing there.) Georg Brandl <georg@python.org> parents: 61859 diff changeset	1254 "cp720",
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1255 "cp737",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1256 "cp775",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1257 "cp850",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1258 "cp852",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1259 "cp855",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1260 "cp856",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1261 "cp857",
61884 c56fb63c648f #8016: add the CP858 codec (approved by Benjamin). (Also add CP720 to the tests, it was missing there.) Georg Brandl <georg@python.org> parents: 61859 diff changeset	1262 "cp858",
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1263 "cp860",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1264 "cp861",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1265 "cp862",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1266 "cp863",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1267 "cp864",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1268 "cp865",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1269 "cp866",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1270 "cp869",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1271 "cp874",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1272 "cp875",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1273 "cp932",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1274 "cp949",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1275 "cp950",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1276 "euc_jis_2004",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1277 "euc_jisx0213",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1278 "euc_jp",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1279 "euc_kr",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1280 "gb18030",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1281 "gb2312",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1282 "gbk",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1283 "hex_codec",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1284 "hp_roman8",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1285 "hz",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1286 "idna",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1287 "iso2022_jp",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1288 "iso2022_jp_1",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1289 "iso2022_jp_2",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1290 "iso2022_jp_2004",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1291 "iso2022_jp_3",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1292 "iso2022_jp_ext",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1293 "iso2022_kr",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1294 "iso8859_1",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1295 "iso8859_10",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1296 "iso8859_11",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1297 "iso8859_13",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1298 "iso8859_14",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1299 "iso8859_15",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1300 "iso8859_16",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1301 "iso8859_2",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1302 "iso8859_3",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1303 "iso8859_4",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1304 "iso8859_5",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1305 "iso8859_6",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1306 "iso8859_7",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1307 "iso8859_8",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1308 "iso8859_9",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1309 "johab",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1310 "koi8_r",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1311 "koi8_u",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1312 "latin_1",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1313 "mac_cyrillic",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1314 "mac_greek",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1315 "mac_iceland",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1316 "mac_latin2",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1317 "mac_roman",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1318 "mac_turkish",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1319 "palmos",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1320 "ptcp154",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1321 "punycode",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1322 "raw_unicode_escape",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1323 "rot_13",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1324 "shift_jis",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1325 "shift_jis_2004",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1326 "shift_jisx0213",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1327 "tis_620",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1328 "unicode_escape",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1329 "unicode_internal",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1330 "utf_16",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1331 "utf_16_be",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1332 "utf_16_le",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1333 "utf_7",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1334 "utf_8",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1335 ]
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1336
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1337 if hasattr(codecs, "mbcs_encode"):
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1338 all_unicode_encodings.append("mbcs")
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1339
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1340 # The following encodings work only with str, not unicode
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1341 all_string_encodings = [
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1342 "quopri_codec",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1343 "string_escape",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1344 "uu_codec",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1345 ]
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1346
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1347 # The following encoding is not tested, because it's not supposed
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1348 # to work:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1349 # "undefined"
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1350
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1351 # The following encodings don't work in stateful mode
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1352 broken_unicode_with_streams = [
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1353 "base64_codec",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1354 "hex_codec",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1355 "punycode",
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1356 "unicode_internal"
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1357 ]
40252 82695e736823 Bug #1586613: fix zlib and bz2 codecs' incremental en/decoders. Georg Brandl <georg@python.org> parents: 40250 diff changeset	1358 broken_incremental_coders = broken_unicode_with_streams[:]
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1359
40271 c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1360 # The following encodings only support "strict" mode
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1361 only_strict_mode = [
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1362 "idna",
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1363 "zlib_codec",
40274 bc8ffc3e3cdb I'm assuming this is correct, it fixes the tests so they pass again Neal Norwitz <nnorwitz@gmail.com> parents: 40271 diff changeset	1364 "bz2_codec",
40271 c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1365 ]
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1366
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1367 try:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1368 import bz2
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1369 except ImportError:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1370 pass
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1371 else:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1372 all_unicode_encodings.append("bz2_codec")
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1373 broken_unicode_with_streams.append("bz2_codec")
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1374
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1375 try:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1376 import zlib
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1377 except ImportError:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1378 pass
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1379 else:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1380 all_unicode_encodings.append("zlib_codec")
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1381 broken_unicode_with_streams.append("zlib_codec")
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1382
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1383 class BasicUnicodeTest(unittest.TestCase):
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1384 def test_basics(self):
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1385 s = u"abc123" # all codecs should be able to encode these
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1386 for encoding in all_unicode_encodings:
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1387 name = codecs.lookup(encoding).name
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1388 if encoding.endswith("_codec"):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1389 name += "_codec"
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1390 elif encoding == "latin_1":
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1391 name = "latin_1"
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1392 self.assertEqual(encoding.replace("_", "-"), name.replace("_", "-"))
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1393 (bytes, size) = codecs.getencoder(encoding)(s)
53855 7dca250343ea Issue 3739: The unicode-internal encoder now reports the number of characters Walter Dörwald <walter@livinglogic.de> parents: 43932 diff changeset	1394 self.assertEqual(size, len(s), "%r != %r (encoding=%r)" % (size, len(s), encoding))
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1395 (chars, size) = codecs.getdecoder(encoding)(bytes)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1396 self.assertEqual(chars, s, "%r != %r (encoding=%r)" % (chars, s, encoding))
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1397
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1398 if encoding not in broken_unicode_with_streams:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1399 # check stream reader/writer
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1400 q = Queue()
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1401 writer = codecs.getwriter(encoding)(q)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1402 encodedresult = ""
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1403 for c in s:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1404 writer.write(c)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1405 encodedresult += q.read()
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1406 q = Queue()
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1407 reader = codecs.getreader(encoding)(q)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1408 decodedresult = u""
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1409 for c in encodedresult:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1410 q.write(c)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1411 decodedresult += reader.read()
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1412 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1413
40252 82695e736823 Bug #1586613: fix zlib and bz2 codecs' incremental en/decoders. Georg Brandl <georg@python.org> parents: 40250 diff changeset	1414 if encoding not in broken_incremental_coders:
37077 38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1415 # check incremental decoder/encoder (fetched via the Python
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1416 # and C API) and iterencode()/iterdecode()
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1417 try:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1418 encoder = codecs.getincrementalencoder(encoding)()
37077 38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1419 cencoder = _testcapi.codec_incrementalencoder(encoding)
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1420 except LookupError: # no IncrementalEncoder
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1421 pass
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1422 else:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1423 # check incremental decoder/encoder
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1424 encodedresult = ""
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1425 for c in s:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1426 encodedresult += encoder.encode(c)
37708 230fff6e0618 Call encode()/decode() with final==True as the last call in the Walter Dörwald <walter@livinglogic.de> parents: 37077 diff changeset	1427 encodedresult += encoder.encode(u"", True)
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1428 decoder = codecs.getincrementaldecoder(encoding)()
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1429 decodedresult = u""
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1430 for c in encodedresult:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1431 decodedresult += decoder.decode(c)
37708 230fff6e0618 Call encode()/decode() with final==True as the last call in the Walter Dörwald <walter@livinglogic.de> parents: 37077 diff changeset	1432 decodedresult += decoder.decode("", True)
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1433 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1434
37077 38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1435 # check C API
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1436 encodedresult = ""
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1437 for c in s:
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1438 encodedresult += cencoder.encode(c)
37708 230fff6e0618 Call encode()/decode() with final==True as the last call in the Walter Dörwald <walter@livinglogic.de> parents: 37077 diff changeset	1439 encodedresult += cencoder.encode(u"", True)
37077 38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1440 cdecoder = _testcapi.codec_incrementaldecoder(encoding)
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1441 decodedresult = u""
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1442 for c in encodedresult:
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1443 decodedresult += cdecoder.decode(c)
37708 230fff6e0618 Call encode()/decode() with final==True as the last call in the Walter Dörwald <walter@livinglogic.de> parents: 37077 diff changeset	1444 decodedresult += cdecoder.decode("", True)
37077 38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1445 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
38400b662404 Add tests for the C APIs PyCodec_IncrementalEncoder() and Walter Dörwald <walter@livinglogic.de> parents: 36996 diff changeset	1446
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1447 # check iterencode()/iterdecode()
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1448 result = u"".join(codecs.iterdecode(codecs.iterencode(s, encoding), encoding))
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1449 self.assertEqual(result, s, "%r != %r (encoding=%r)" % (result, s, encoding))
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1450
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1451 # check iterencode()/iterdecode() with empty string
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1452 result = u"".join(codecs.iterdecode(codecs.iterencode(u"", encoding), encoding))
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1453 self.assertEqual(result, u"")
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1454
40271 c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1455 if encoding not in only_strict_mode:
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1456 # check incremental decoder/encoder with errors argument
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1457 try:
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1458 encoder = codecs.getincrementalencoder(encoding)("ignore")
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1459 cencoder = _testcapi.codec_incrementalencoder(encoding, "ignore")
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1460 except LookupError: # no IncrementalEncoder
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1461 pass
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1462 else:
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1463 encodedresult = "".join(encoder.encode(c) for c in s)
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1464 decoder = codecs.getincrementaldecoder(encoding)("ignore")
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1465 decodedresult = u"".join(decoder.decode(c) for c in encodedresult)
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1466 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
40284 ff678669aa6e Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 40274 diff changeset	1467
40271 c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1468 encodedresult = "".join(cencoder.encode(c) for c in s)
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1469 cdecoder = _testcapi.codec_incrementaldecoder(encoding, "ignore")
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1470 decodedresult = u"".join(cdecoder.decode(c) for c in encodedresult)
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1471 self.assertEqual(decodedresult, s, "%r != %r (encoding=%r)" % (decodedresult, s, encoding))
c5a2f431c698 Add tests for incremental codecs with an errors Walter Dörwald <walter@livinglogic.de> parents: 40251 diff changeset	1472
34750 679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1473 def test_seek(self):
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1474 # all codecs should be able to encode these
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1475 s = u"%s\n%s\n" % (100u"abc123", 100u"def456")
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1476 for encoding in all_unicode_encodings:
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1477 if encoding == "idna": # FIXME: See SF bug #1163178
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1478 continue
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1479 if encoding in broken_unicode_with_streams:
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1480 continue
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1481 reader = codecs.getreader(encoding)(StringIO.StringIO(s.encode(encoding)))
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1482 for t in xrange(5):
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1483 # Test that calling seek resets the internal codec state and buffers
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1484 reader.seek(0, 0)
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1485 line = reader.readline()
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1486 self.assertEqual(s[:len(line)], line)
679c16115613 Reset internal buffers when seek() is called. This fixes SF bug #1156259. Walter Dörwald <walter@livinglogic.de> parents: 34562 diff changeset	1487
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1488 def test_bad_decode_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1489 for encoding in all_unicode_encodings:
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1490 decoder = codecs.getdecoder(encoding)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1491 self.assertRaises(TypeError, decoder)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1492 if encoding not in ("idna", "punycode"):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1493 self.assertRaises(TypeError, decoder, 42)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1494
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1495 def test_bad_encode_args(self):
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1496 for encoding in all_unicode_encodings:
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1497 encoder = codecs.getencoder(encoding)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1498 self.assertRaises(TypeError, encoder)
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1499
38787 70b166ca1c49 Verify the crash due to EncodingMap not initialized does not return Neal Norwitz <nnorwitz@gmail.com> parents: 37719 diff changeset	1500 def test_encoding_map_type_initialized(self):
70b166ca1c49 Verify the crash due to EncodingMap not initialized does not return Neal Norwitz <nnorwitz@gmail.com> parents: 37719 diff changeset	1501 from encodings import cp1140
70b166ca1c49 Verify the crash due to EncodingMap not initialized does not return Neal Norwitz <nnorwitz@gmail.com> parents: 37719 diff changeset	1502 # This used to crash, we are only verifying there's no crash.
70b166ca1c49 Verify the crash due to EncodingMap not initialized does not return Neal Norwitz <nnorwitz@gmail.com> parents: 37719 diff changeset	1503 table_type = type(cp1140.encoding_table)
70b166ca1c49 Verify the crash due to EncodingMap not initialized does not return Neal Norwitz <nnorwitz@gmail.com> parents: 37719 diff changeset	1504 self.assertEqual(table_type, table_type)
70b166ca1c49 Verify the crash due to EncodingMap not initialized does not return Neal Norwitz <nnorwitz@gmail.com> parents: 37719 diff changeset	1505
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1506 class BasicStrTest(unittest.TestCase):
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1507 def test_basics(self):
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1508 s = "abc123"
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1509 for encoding in all_string_encodings:
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1510 (bytes, size) = codecs.getencoder(encoding)(s)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1511 self.assertEqual(size, len(s))
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1512 (chars, size) = codecs.getdecoder(encoding)(bytes)
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1513 self.assertEqual(chars, s, "%r != %r (encoding=%r)" % (chars, s, encoding))
b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1514
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1515 class CharmapTest(unittest.TestCase):
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1516 def test_decode_with_string_map(self):
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1517 self.assertEqual(
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1518 codecs.charmap_decode("\x00\x01\x02", "strict", u"abc"),
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1519 (u"abc", 3)
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1520 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1521
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1522 self.assertEqual(
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1523 codecs.charmap_decode("\x00\x01\x02", "replace", u"ab"),
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1524 (u"ab\ufffd", 3)
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1525 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1526
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1527 self.assertEqual(
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1528 codecs.charmap_decode("\x00\x01\x02", "replace", u"ab\ufffe"),
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1529 (u"ab\ufffd", 3)
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1530 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1531
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1532 self.assertEqual(
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1533 codecs.charmap_decode("\x00\x01\x02", "ignore", u"ab"),
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1534 (u"ab", 3)
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1535 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1536
80478 c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1537 self.assertRaises(UnicodeDecodeError,
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1538 codecs.charmap_decode, b"\x00\x01\x02", "strict", u"ab"
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1539 )
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1540
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1541 self.assertEqual(
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1542 codecs.charmap_decode("\x00\x01\x02", "ignore", u"ab\ufffe"),
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1543 (u"ab", 3)
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1544 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1545
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1546 allbytes = "".join(chr(i) for i in xrange(256))
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1547 self.assertEqual(
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1548 codecs.charmap_decode(allbytes, "ignore", u""),
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1549 (u"", len(allbytes))
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1550 )
22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1551
80478 c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1552 def test_decode_with_int2str_map(self):
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1553 self.assertEqual(
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1554 codecs.charmap_decode("\x00\x01\x02", "strict",
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1555 {0: u'a', 1: u'b', 2: u'c'}),
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1556 (u"abc", 3)
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1557 )
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1558
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1559 self.assertEqual(
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1560 codecs.charmap_decode("\x00\x01\x02", "strict",
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1561 {0: u'Aa', 1: u'Bb', 2: u'Cc'}),
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1562 (u"AaBbCc", 3)
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1563 )
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1564
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1565 self.assertEqual(
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1566 codecs.charmap_decode("\x00\x01\x02", "strict",
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1567 {0: u'\U0010FFFF', 1: u'b', 2: u'c'}),
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1568 (u"\U0010FFFFbc", 3)
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1569 )
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1570
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1571 self.assertEqual(
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1572 codecs.charmap_decode("\x00\x01\x02", "strict",
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1573 {0: u'a', 1: u'b', 2: u''}),
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1574 (u"ab", 3)
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1575 )
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1576
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1577 self.assertRaises(UnicodeDecodeError,
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1578 codecs.charmap_decode, "\x00\x01\x02", "strict",
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1579 {0: u'a', 1: u'b'}
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1580 )
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1581
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1582 self.assertEqual(
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1583 codecs.charmap_decode("\x00\x01\x02", "replace",
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1584 {0: u'a', 1: u'b'}),
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1585 (u"ab\ufffd", 3)
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1586 )
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1587
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1588 self.assertEqual(
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1589 codecs.charmap_decode("\x00\x01\x02", "replace",
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1590 {0: u'a', 1: u'b', 2: None}),
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1591 (u"ab\ufffd", 3)
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1592 )
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1593
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1594 self.assertEqual(
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1595 codecs.charmap_decode("\x00\x01\x02", "ignore",
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1596 {0: u'a', 1: u'b'}),
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1597 (u"ab", 3)
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1598 )
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1599
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1600 self.assertEqual(
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1601 codecs.charmap_decode("\x00\x01\x02", "ignore",
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1602 {0: u'a', 1: u'b', 2: None}),
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1603 (u"ab", 3)
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1604 )
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1605
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1606 allbytes = bytes(range(256))
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1607 self.assertEqual(
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1608 codecs.charmap_decode(allbytes, "ignore", {}),
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1609 (u"", len(allbytes))
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1610 )
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1611
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1612 def test_decode_with_int2int_map(self):
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1613 a = ord(u'a')
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1614 b = ord(u'b')
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1615 c = ord(u'c')
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1616
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1617 self.assertEqual(
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1618 codecs.charmap_decode("\x00\x01\x02", "strict",
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1619 {0: a, 1: b, 2: c}),
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1620 (u"abc", 3)
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1621 )
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1622
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1623 # Issue #15379
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1624 self.assertEqual(
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1625 codecs.charmap_decode("\x00\x01\x02", "strict",
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1626 {0: 0x10FFFF, 1: b, 2: c}),
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1627 (u"\U0010FFFFbc", 3)
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1628 )
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1629
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1630 self.assertRaises(TypeError,
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1631 codecs.charmap_decode, "\x00\x01\x02", "strict",
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1632 {0: 0x110000, 1: b, 2: c}
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1633 )
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1634
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1635 self.assertRaises(UnicodeDecodeError,
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1636 codecs.charmap_decode, "\x00\x01\x02", "strict",
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1637 {0: a, 1: b},
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1638 )
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1639
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1640 self.assertEqual(
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1641 codecs.charmap_decode("\x00\x01\x02", "replace",
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1642 {0: a, 1: b}),
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1643 (u"ab\ufffd", 3)
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1644 )
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1645
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1646 self.assertEqual(
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1647 codecs.charmap_decode("\x00\x01\x02", "ignore",
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1648 {0: a, 1: b}),
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1649 (u"ab", 3)
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1650 )
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1651
c7ce91756472 Issue #15379: Fix passing of non-BMP characters as integers for the charmap decoder (already working as unicode strings). Antoine Pitrou <solipsis@pitrou.net> parents: 78208 diff changeset	1652
40240 d245efc7ec89 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1653 class WithStmtTest(unittest.TestCase):
d245efc7ec89 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1654 def test_encodedfile(self):
d245efc7ec89 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1655 f = StringIO.StringIO("\xc3\xbc")
d245efc7ec89 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1656 with codecs.EncodedFile(f, "latin-1", "utf-8") as ef:
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1657 self.assertEqual(ef.read(), "\xfc")
40240 d245efc7ec89 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1658
d245efc7ec89 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1659 def test_streamreaderwriter(self):
d245efc7ec89 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1660 f = StringIO.StringIO("\xc3\xbc")
d245efc7ec89 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1661 info = codecs.lookup("utf-8")
d245efc7ec89 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1662 with codecs.StreamReaderWriter(f, info.streamreader,
d245efc7ec89 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1663 info.streamwriter, 'strict') as srw:
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1664 self.assertEqual(srw.read(), u"\xfc")
40240 d245efc7ec89 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1665
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1666
61848 2a389734d570 Merged revisions 81459 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 59169 diff changeset	1667 class BomTest(unittest.TestCase):
2a389734d570 Merged revisions 81459 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 59169 diff changeset	1668 def test_seek0(self):
61861 ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1669 data = u"1234567890"
61848 2a389734d570 Merged revisions 81459 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 59169 diff changeset	1670 tests = ("utf-16",
2a389734d570 Merged revisions 81459 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 59169 diff changeset	1671 "utf-16-le",
2a389734d570 Merged revisions 81459 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 59169 diff changeset	1672 "utf-16-be",
2a389734d570 Merged revisions 81459 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 59169 diff changeset	1673 "utf-32",
2a389734d570 Merged revisions 81459 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 59169 diff changeset	1674 "utf-32-le",
2a389734d570 Merged revisions 81459 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 59169 diff changeset	1675 "utf-32-be")
70295 10c7fac542e6 test_codecs now removes the temporay file (created by the test) Victor Stinner <victor.stinner@haypocalc.com> parents: 68256 diff changeset	1676 self.addCleanup(test_support.unlink, test_support.TESTFN)
61848 2a389734d570 Merged revisions 81459 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 59169 diff changeset	1677 for encoding in tests:
61861 ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1678 # Check if the BOM is written only once
ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1679 with codecs.open(test_support.TESTFN, 'w+', encoding=encoding) as f:
61848 2a389734d570 Merged revisions 81459 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 59169 diff changeset	1680 f.write(data)
2a389734d570 Merged revisions 81459 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 59169 diff changeset	1681 f.write(data)
2a389734d570 Merged revisions 81459 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 59169 diff changeset	1682 f.seek(0)
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1683 self.assertEqual(f.read(), data * 2)
61848 2a389734d570 Merged revisions 81459 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 59169 diff changeset	1684 f.seek(0)
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1685 self.assertEqual(f.read(), data * 2)
61848 2a389734d570 Merged revisions 81459 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 59169 diff changeset	1686
61861 ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1687 # Check that the BOM is written after a seek(0)
ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1688 with codecs.open(test_support.TESTFN, 'w+', encoding=encoding) as f:
ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1689 f.write(data[0])
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1690 self.assertNotEqual(f.tell(), 0)
61861 ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1691 f.seek(0)
ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1692 f.write(data)
ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1693 f.seek(0)
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1694 self.assertEqual(f.read(), data)
61861 ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1695
ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1696 # (StreamWriter) Check that the BOM is written after a seek(0)
ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1697 with codecs.open(test_support.TESTFN, 'w+', encoding=encoding) as f:
ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1698 f.writer.write(data[0])
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1699 self.assertNotEqual(f.writer.tell(), 0)
61861 ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1700 f.writer.seek(0)
ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1701 f.writer.write(data)
ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1702 f.seek(0)
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1703 self.assertEqual(f.read(), data)
61861 ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1704
ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1705 # Check that the BOM is not written after a seek() at a position
ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1706 # different than the start
ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1707 with codecs.open(test_support.TESTFN, 'w+', encoding=encoding) as f:
ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1708 f.write(data)
ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1709 f.seek(f.tell())
ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1710 f.write(data)
ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1711 f.seek(0)
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1712 self.assertEqual(f.read(), data * 2)
61861 ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1713
ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1714 # (StreamWriter) Check that the BOM is not written after a seek()
ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1715 # at a position different than the start
ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1716 with codecs.open(test_support.TESTFN, 'w+', encoding=encoding) as f:
ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1717 f.writer.write(data)
ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1718 f.writer.seek(f.writer.tell())
ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1719 f.writer.write(data)
ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1720 f.seek(0)
66348 614c84670c63 Merged revisions 86596 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 62255 diff changeset	1721 self.assertEqual(f.read(), data * 2)
61861 ab082e387dd5 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61848 diff changeset	1722
61848 2a389734d570 Merged revisions 81459 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 59169 diff changeset	1723
20013 e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	1724 def test_main():
28917 1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	1725 test_support.run_unittest(
42649 c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	1726 UTF32Test,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	1727 UTF32LETest,
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 41445 diff changeset	1728 UTF32BETest,
28917 1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	1729 UTF16Test,
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	1730 UTF16LETest,
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	1731 UTF16BETest,
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32572 diff changeset	1732 UTF8Test,
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35889 diff changeset	1733 UTF8SigTest,
35882 3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1734 UTF7Test,
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1735 UTF16ExTest,
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1736 ReadBufferTest,
3236b5fb1fb7 Add tests for various error cases and for readbuffer_encode() and Walter Dörwald <walter@livinglogic.de> parents: 35702 diff changeset	1737 CharBufferTest,
28917 1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	1738 EscapeDecodeTest,
1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	1739 RecodingTest,
1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	1740 PunycodeTest,
35484 4fc7a7edcb4b SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain Walter Dörwald <walter@livinglogic.de> parents: 35435 diff changeset	1741 UnicodeInternalTest,
31795 bf37429f6d9f Add test case for unicode(somestring, "idna"). Martin v. Löwis <martin@v.loewis.de> parents: 28917 diff changeset	1742 NameprepTest,
37719 cf71c3814b0b Add a BufferedIncrementalEncoder class that can be used for implementing Walter Dörwald <walter@livinglogic.de> parents: 37708 diff changeset	1743 IDNACodecTest,
33782 a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	1744 CodecsModuleTest,
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1745 StreamReaderTest,
40240 d245efc7ec89 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1746 EncodedFileTest,
35269 2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34925 diff changeset	1747 Str2StrTest,
34279 b3af7c6db1c6 Add a test that checks the basic functionality of every encoding. Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	1748 BasicUnicodeTest,
35702 22a720b35314 Part of SF patch #1313939: Speedup charmap decoding by extending Walter Dörwald <walter@livinglogic.de> parents: 35484 diff changeset	1749 BasicStrTest,
40240 d245efc7ec89 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1750 CharmapTest,
d245efc7ec89 Fix codecs.EncodedFile which did not use file_encoding in 2.5.0, and Georg Brandl <georg@python.org> parents: 38787 diff changeset	1751 WithStmtTest,
61848 2a389734d570 Merged revisions 81459 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 59169 diff changeset	1752 BomTest,
28917 1741af48fd9f Combine the functionality of test_support.run_unittest() Walter Dörwald <walter@livinglogic.de> parents: 28774 diff changeset	1753 )
20013 e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	1754
e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	1755
e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	1756 if __name__ == "__main__":
e3184e46303b Change the PyUnit-based tests to use the test_main() approach. This Fred Drake <fdrake@acm.org> parents: 18163 diff changeset	1757 test_main()

Mercurial > cpython-lru_cache-weakref

annotate Lib/test/test_codecs.py @ 80661:94de77bd0b4b 2.7