cpython-lru_cache-weakref: Lib/codecs.py annotate

annotate Lib/codecs.py @ 87690:66c1c9f32567 default tip

Add use_weakref option to functools.lru_cache The use_weakref option makes lru_cache create weak references to the function arguments when creating the cache key, which prevents them from staying in the memory forever even after all other references to them are gone. This is a proof of concept for http://bugs.python.org/issue19859

author	Radomir Dopieralski <devel@sheep.art.pl>
date	Tue, 03 Dec 2013 16:02:48 +0100
parents	d68df99d7a57
children

rev	line source
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	1 """ codecs -- Python Codec Registry, API and helpers.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	2
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	3
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	4 Written by Marc-Andre Lemburg (mal@lemburg.com).
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	5
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	6 (c) Copyright CNRI, All Rights Reserved. NO WARRANTY.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	7
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	8 """#"
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	9
44098 0889daf6d0d9 #1535: rename __builtin__ module to builtins. Georg Brandl <georg@python.org> parents: 41606 diff changeset	10 import builtins, sys
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	11
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	12 ### Registry and builtin stateless codec functions
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	13
12195 bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12051 diff changeset	14 try:
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12051 diff changeset	15 from _codecs import *
40587 6b18a095c2e7 SF patch 1631942 by Collin Winter: Guido van Rossum <guido@python.org> parents: 40450 diff changeset	16 except ImportError as why:
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	17 raise SystemError('Failed to load the builtin codecs: %s' % why)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	18
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	19 __all__ = ["register", "lookup", "open", "EncodedFile", "BOM", "BOM_BE",
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	20 "BOM_LE", "BOM32_BE", "BOM32_LE", "BOM64_BE", "BOM64_LE",
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	21 "BOM_UTF8", "BOM_UTF16", "BOM_UTF16_LE", "BOM_UTF16_BE",
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	22 "BOM_UTF32", "BOM_UTF32_LE", "BOM_UTF32_BE",
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	23 "strict_errors", "ignore_errors", "replace_errors",
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	24 "xmlcharrefreplace_errors",
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	25 "register_error", "lookup_error"]
16070 b5345e99596b added __all__ lists to a number of Python modules Skip Montanaro <skip@pobox.com> parents: 15867 diff changeset	26
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	27 ### Constants
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	28
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	29 #
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	30 # Byte Order Mark (BOM = ZERO WIDTH NO-BREAK SPACE = U+FEFF)
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	31 # and its possible byte string values
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	32 # for UTF8/UTF16/UTF32 output and little/big endian machines
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	33 #
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	34
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	35 # UTF-8
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	36 BOM_UTF8 = b'\xef\xbb\xbf'
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	37
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	38 # UTF-16, little endian
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	39 BOM_LE = BOM_UTF16_LE = b'\xff\xfe'
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	40
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	41 # UTF-16, big endian
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	42 BOM_BE = BOM_UTF16_BE = b'\xfe\xff'
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	43
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	44 # UTF-32, little endian
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	45 BOM_UTF32_LE = b'\xff\xfe\x00\x00'
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	46
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	47 # UTF-32, big endian
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	48 BOM_UTF32_BE = b'\x00\x00\xfe\xff'
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	49
26616 20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	50 if sys.byteorder == 'little':
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	51
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	52 # UTF-16, native endianness
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	53 BOM = BOM_UTF16 = BOM_UTF16_LE
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	54
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	55 # UTF-32, native endianness
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	56 BOM_UTF32 = BOM_UTF32_LE
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	57
26616 20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	58 else:
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	59
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	60 # UTF-16, native endianness
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	61 BOM = BOM_UTF16 = BOM_UTF16_BE
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	62
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	63 # UTF-32, native endianness
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	64 BOM_UTF32 = BOM_UTF32_BE
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	65
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	66 # Old broken names (don't use in new code)
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	67 BOM32_LE = BOM_UTF16_LE
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	68 BOM32_BE = BOM_UTF16_BE
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	69 BOM64_LE = BOM_UTF32_LE
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	70 BOM64_BE = BOM_UTF32_BE
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	71
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	72
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	73 ### Codec base classes (defining the API)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	74
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	75 class CodecInfo(tuple):
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 84165 diff changeset	76 """Codec details when looking up the codec registry"""
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 84165 diff changeset	77
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 84165 diff changeset	78 # Private API to allow Python 3.4 to blacklist the known non-Unicode
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 84165 diff changeset	79 # codecs in the standard library. A more general mechanism to
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 84165 diff changeset	80 # reliably distinguish test encodings from other codecs will hopefully
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 84165 diff changeset	81 # be defined for Python 3.5
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 84165 diff changeset	82 #
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 84165 diff changeset	83 # See http://bugs.python.org/issue19619
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 84165 diff changeset	84 _is_text_encoding = True # Assume codecs are text encodings by default
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	85
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	86 def __new__(cls, encode, decode, streamreader=None, streamwriter=None,
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 84165 diff changeset	87 incrementalencoder=None, incrementaldecoder=None, name=None,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 84165 diff changeset	88 *, _is_text_encoding=None):
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	89 self = tuple.__new__(cls, (encode, decode, streamreader, streamwriter))
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	90 self.name = name
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	91 self.encode = encode
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	92 self.decode = decode
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	93 self.incrementalencoder = incrementalencoder
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	94 self.incrementaldecoder = incrementaldecoder
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	95 self.streamwriter = streamwriter
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	96 self.streamreader = streamreader
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 84165 diff changeset	97 if _is_text_encoding is not None:
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 84165 diff changeset	98 self._is_text_encoding = _is_text_encoding
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	99 return self
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	100
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	101 def __repr__(self):
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	102 return "<%s.%s object for encoding %s at 0x%x>" % \
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	103 (self.__class__.__module__, self.__class__.__name__,
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	104 self.name, id(self))
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	105
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	106 class Codec:
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	107
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	108 """ Defines the interface for stateless encoders/decoders.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	109
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	110 The .encode()/.decode() methods may use different error
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	111 handling schemes by providing the errors argument. These
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	112 string values are predefined:
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	113
12051 d89556c04ef9 Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12011 diff changeset	114 'strict' - raise a ValueError error (or a subclass)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	115 'ignore' - ignore the character and continue with the next
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	116 'replace' - replace with a suitable replacement character;
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	117 Python will use the official U+FFFD REPLACEMENT
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	118 CHARACTER for the builtin Unicode codecs on
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	119 decoding and '?' on encoding.
84165 55f611f55952 Describe 'surrogateescape' in the documentation. Andrew Kuchling <amk@amk.ca> parents: 79363 diff changeset	120 'surrogateescape' - replace with private codepoints U+DCnn.
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	121 'xmlcharrefreplace' - Replace with the appropriate XML
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	122 character reference (only for encoding).
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	123 'backslashreplace' - Replace with backslashed escape sequences
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	124 (only for encoding).
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	125
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	126 The set of allowed values can be extended via register_error.
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	127
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	128 """
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	129 def encode(self, input, errors='strict'):
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	130
12010 bd1014215f62 "intput" --> "input" (in docstring); noted by Detlef Lannert Fred Drake <fdrake@acm.org> parents: 11990 diff changeset	131 """ Encodes the object input and returns a tuple (output
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	132 object, length consumed).
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	133
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	134 errors defines the error handling to apply. It defaults to
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	135 'strict' handling.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	136
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	137 The method may not store state in the Codec instance. Use
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	138 StreamCodec for codecs which have to keep state in order to
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	139 make encoding/decoding efficient.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	140
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	141 The encoder must be able to handle zero length input and
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	142 return an empty object of the output object type in this
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	143 situation.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	144
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	145 """
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	146 raise NotImplementedError
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	147
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	148 def decode(self, input, errors='strict'):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	149
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	150 """ Decodes the object input and returns a tuple (output
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	151 object, length consumed).
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	152
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	153 input must be an object which provides the bf_getreadbuf
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	154 buffer slot. Python strings, buffer objects and memory
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	155 mapped files are examples of objects providing this slot.
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	156
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	157 errors defines the error handling to apply. It defaults to
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	158 'strict' handling.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	159
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	160 The method may not store state in the Codec instance. Use
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	161 StreamCodec for codecs which have to keep state in order to
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	162 make encoding/decoding efficient.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	163
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	164 The decoder must be able to handle zero length input and
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	165 return an empty object of the output object type in this
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	166 situation.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	167
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	168 """
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	169 raise NotImplementedError
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	170
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	171 class IncrementalEncoder(object):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	172 """
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	173 An IncrementalEncoder encodes an input in multiple steps. The input can
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	174 be passed piece by piece to the encode() method. The IncrementalEncoder
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	175 remembers the state of the encoding process between calls to encode().
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	176 """
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	177 def __init__(self, errors='strict'):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	178 """
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	179 Creates an IncrementalEncoder instance.
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	180
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	181 The IncrementalEncoder may use different error handling schemes by
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	182 providing the errors keyword argument. See the module docstring
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	183 for a list of possible values.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	184 """
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	185 self.errors = errors
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	186 self.buffer = ""
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	187
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	188 def encode(self, input, final=False):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	189 """
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	190 Encodes input and returns the resulting object.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	191 """
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	192 raise NotImplementedError
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	193
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	194 def reset(self):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	195 """
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	196 Resets the encoder to the initial state.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	197 """
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	198
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	199 def getstate(self):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	200 """
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	201 Return the current state of the encoder.
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	202 """
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	203 return 0
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	204
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	205 def setstate(self, state):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	206 """
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	207 Set the current state of the encoder. state must have been
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	208 returned by getstate().
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	209 """
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	210
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	211 class BufferedIncrementalEncoder(IncrementalEncoder):
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	212 """
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	213 This subclass of IncrementalEncoder can be used as the baseclass for an
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	214 incremental encoder if the encoder must keep some of the output in a
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	215 buffer between calls to encode().
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	216 """
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	217 def __init__(self, errors='strict'):
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	218 IncrementalEncoder.__init__(self, errors)
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	219 # unencoded input that is kept between calls to encode()
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	220 self.buffer = ""
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	221
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	222 def _buffer_encode(self, input, errors, final):
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	223 # Overwrite this method in subclasses: It must encode input
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	224 # and return an (output, length consumed) tuple
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	225 raise NotImplementedError
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	226
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	227 def encode(self, input, final=False):
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	228 # encode input (taking the buffer into account)
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	229 data = self.buffer + input
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	230 (result, consumed) = self._buffer_encode(data, self.errors, final)
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	231 # keep unencoded input until the next call
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	232 self.buffer = data[consumed:]
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	233 return result
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	234
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	235 def reset(self):
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	236 IncrementalEncoder.reset(self)
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	237 self.buffer = ""
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	238
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	239 def getstate(self):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	240 return self.buffer or 0
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	241
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	242 def setstate(self, state):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	243 self.buffer = state or ""
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	244
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	245 class IncrementalDecoder(object):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	246 """
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	247 An IncrementalDecoder decodes an input in multiple steps. The input can
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	248 be passed piece by piece to the decode() method. The IncrementalDecoder
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	249 remembers the state of the decoding process between calls to decode().
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	250 """
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	251 def __init__(self, errors='strict'):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	252 """
45835 04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	253 Create a IncrementalDecoder instance.
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	254
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	255 The IncrementalDecoder may use different error handling schemes by
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	256 providing the errors keyword argument. See the module docstring
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	257 for a list of possible values.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	258 """
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	259 self.errors = errors
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	260
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	261 def decode(self, input, final=False):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	262 """
45835 04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	263 Decode input and returns the resulting object.
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	264 """
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	265 raise NotImplementedError
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	266
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	267 def reset(self):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	268 """
45835 04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	269 Reset the decoder to the initial state.
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	270 """
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	271
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	272 def getstate(self):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	273 """
45835 04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	274 Return the current state of the decoder.
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	275
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	276 This must be a (buffered_input, additional_state_info) tuple.
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	277 buffered_input must be a bytes object containing bytes that
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	278 were passed to decode() that have not yet been converted.
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	279 additional_state_info must be a non-negative integer
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	280 representing the state of the decoder WITHOUT yet having
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	281 processed the contents of buffered_input. In the initial state
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	282 and after reset(), getstate() must return (b"", 0).
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	283 """
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	284 return (b"", 0)
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	285
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	286 def setstate(self, state):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	287 """
45835 04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	288 Set the current state of the decoder.
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	289
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	290 state must have been returned by getstate(). The effect of
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	291 setstate((b"", 0)) must be equivalent to reset().
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	292 """
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	293
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	294 class BufferedIncrementalDecoder(IncrementalDecoder):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	295 """
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	296 This subclass of IncrementalDecoder can be used as the baseclass for an
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	297 incremental decoder if the decoder must be able to handle incomplete
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	298 byte sequences.
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	299 """
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	300 def __init__(self, errors='strict'):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	301 IncrementalDecoder.__init__(self, errors)
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	302 # undecoded input that is kept between calls to decode()
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	303 self.buffer = b""
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	304
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	305 def _buffer_decode(self, input, errors, final):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	306 # Overwrite this method in subclasses: It must decode input
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	307 # and return an (output, length consumed) tuple
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	308 raise NotImplementedError
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	309
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	310 def decode(self, input, final=False):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	311 # decode input (taking the buffer into account)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	312 data = self.buffer + input
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	313 (result, consumed) = self._buffer_decode(data, self.errors, final)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	314 # keep undecoded input until the next call
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	315 self.buffer = data[consumed:]
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	316 return result
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	317
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	318 def reset(self):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	319 IncrementalDecoder.reset(self)
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	320 self.buffer = b""
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	321
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	322 def getstate(self):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	323 # additional state info is always 0
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	324 return (self.buffer, 0)
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	325
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	326 def setstate(self, state):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	327 # ignore additional state info
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	328 self.buffer = state[0]
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	329
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	330 #
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	331 # The StreamWriter and StreamReader class provide generic working
19967 0b51e607f333 Fix typo in comment Andrew M. Kuchling <amk@amk.ca> parents: 18936 diff changeset	332 # interfaces which can be used to implement new encoding submodules
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	333 # very easily. See encodings/utf_8.py for an example on how this is
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	334 # done.
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	335 #
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	336
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	337 class StreamWriter(Codec):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	338
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	339 def __init__(self, stream, errors='strict'):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	340
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	341 """ Creates a StreamWriter instance.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	342
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	343 stream must be a file-like object open for writing
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	344 (binary) data.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	345
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	346 The StreamWriter may use different error handling
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	347 schemes by providing the errors keyword argument. These
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	348 parameters are predefined:
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	349
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	350 'strict' - raise a ValueError (or a subclass)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	351 'ignore' - ignore the character and continue with the next
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	352 'replace'- replace with a suitable replacement character
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	353 'xmlcharrefreplace' - Replace with the appropriate XML
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	354 character reference.
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	355 'backslashreplace' - Replace with backslashed escape
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	356 sequences (only for encoding).
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	357
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	358 The set of allowed parameter values can be extended via
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	359 register_error.
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	360 """
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	361 self.stream = stream
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	362 self.errors = errors
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	363
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	364 def write(self, object):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	365
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	366 """ Writes the object's contents encoded to self.stream.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	367 """
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	368 data, consumed = self.encode(object, self.errors)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	369 self.stream.write(data)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	370
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	371 def writelines(self, list):
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	372
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	373 """ Writes the concatenated list of strings to the stream
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	374 using .write().
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	375 """
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	376 self.write(''.join(list))
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	377
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	378 def reset(self):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	379
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	380 """ Flushes and resets the codec buffers used for keeping state.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	381
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	382 Calling this method should ensure that the data on the
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	383 output is put into a clean state, that allows appending
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	384 of new fresh data without having to rescan the whole
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	385 stream to recover state.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	386
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	387 """
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	388 pass
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	389
61862 830cd89b01e8 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61849 diff changeset	390 def seek(self, offset, whence=0):
830cd89b01e8 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61849 diff changeset	391 self.stream.seek(offset, whence)
830cd89b01e8 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61849 diff changeset	392 if whence == 0 and offset == 0:
830cd89b01e8 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61849 diff changeset	393 self.reset()
830cd89b01e8 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61849 diff changeset	394
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	395 def __getattr__(self, name,
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	396 getattr=getattr):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	397
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	398 """ Inherit all other methods from the underlying stream.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	399 """
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	400 return getattr(self.stream, name)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	401
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	402 def __enter__(self):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	403 return self
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	404
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	405 def __exit__(self, type, value, tb):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	406 self.stream.close()
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	407
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	408 ###
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	409
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	410 class StreamReader(Codec):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	411
66605 7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 61862 diff changeset	412 charbuffertype = str
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 61862 diff changeset	413
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	414 def __init__(self, stream, errors='strict'):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	415
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	416 """ Creates a StreamReader instance.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	417
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	418 stream must be a file-like object open for reading
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	419 (binary) data.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	420
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	421 The StreamReader may use different error handling
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	422 schemes by providing the errors keyword argument. These
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	423 parameters are predefined:
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	424
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	425 'strict' - raise a ValueError (or a subclass)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	426 'ignore' - ignore the character and continue with the next
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	427 'replace'- replace with a suitable replacement character;
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	428
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	429 The set of allowed parameter values can be extended via
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	430 register_error.
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	431 """
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	432 self.stream = stream
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	433 self.errors = errors
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	434 self.bytebuffer = b""
66605 7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 61862 diff changeset	435 self._empty_charbuffer = self.charbuffertype()
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 61862 diff changeset	436 self.charbuffer = self._empty_charbuffer
35589 5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	437 self.linebuffer = None
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	438
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	439 def decode(self, input, errors='strict'):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	440 raise NotImplementedError
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	441
35415 ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	442 def read(self, size=-1, chars=-1, firstline=False):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	443
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	444 """ Decodes data from the stream self.stream and returns the
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	445 resulting object.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	446
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	447 chars indicates the number of characters to read from the
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	448 stream. read() will never return more than chars
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	449 characters, but it might return less, if there are not enough
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	450 characters available.
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	451
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	452 size indicates the approximate maximum number of bytes to
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	453 read from the stream for decoding purposes. The decoder
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	454 can modify this setting as appropriate. The default value
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	455 -1 indicates to read and decode as much as possible. size
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	456 is intended to prevent having to decode huge files in one
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	457 step.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	458
35415 ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	459 If firstline is true, and a UnicodeDecodeError happens
ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	460 after the first line terminator in the input only the first line
ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	461 will be returned, the rest of the input will be kept until the
ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	462 next call to read().
ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	463
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	464 The method should use a greedy read strategy meaning that
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	465 it should read as much data as is allowed within the
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	466 definition of the encoding and the given size, e.g. if
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	467 optional encoding endings or state markers are available
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	468 on the stream, these should be read too.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	469 """
35589 5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	470 # If we have lines cached, first merge them back into characters
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	471 if self.linebuffer:
66605 7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 61862 diff changeset	472 self.charbuffer = self._empty_charbuffer.join(self.linebuffer)
35589 5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	473 self.linebuffer = None
36121 686306a67e9b Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 35589 diff changeset	474
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	475 # read until we get the required number of characters (if available)
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	476 while True:
79363 bb77400af434 Issue16097 Fix small typo in comment (patch by Wael Al Jishi) Tim Golden <mail@timgolden.me.uk> parents: 77471 diff changeset	477 # can the request be satisfied from the character buffer?
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	478 if chars < 0:
36855 204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	479 if size < 0:
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	480 if self.charbuffer:
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	481 break
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	482 elif len(self.charbuffer) >= size:
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	483 break
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	484 else:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	485 if len(self.charbuffer) >= chars:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	486 break
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	487 # we need more data
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	488 if size < 0:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	489 newdata = self.stream.read()
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	490 else:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	491 newdata = self.stream.read(size)
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	492 # decode bytes (those remaining from the last call included)
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	493 data = self.bytebuffer + newdata
35415 ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	494 try:
ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	495 newchars, decodedbytes = self.decode(data, self.errors)
40587 6b18a095c2e7 SF patch 1631942 by Collin Winter: Guido van Rossum <guido@python.org> parents: 40450 diff changeset	496 except UnicodeDecodeError as exc:
35415 ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	497 if firstline:
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	498 newchars, decodedbytes = \
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	499 self.decode(data[:exc.start], self.errors)
72485 83f43b58c988 #13012: use splitlines(keepends=True/False) instead of splitlines(0/1). Ezio Melotti <ezio.melotti@gmail.com> parents: 70437 diff changeset	500 lines = newchars.splitlines(keepends=True)
35415 ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	501 if len(lines)<=1:
ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	502 raise
ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	503 else:
ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	504 raise
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	505 # keep undecoded bytes until the next call
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	506 self.bytebuffer = data[decodedbytes:]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	507 # put new characters in the character buffer
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	508 self.charbuffer += newchars
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	509 # there was no data available
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	510 if not newdata:
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	511 break
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	512 if chars < 0:
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	513 # Return everything we've got
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	514 result = self.charbuffer
66605 7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 61862 diff changeset	515 self.charbuffer = self._empty_charbuffer
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	516 else:
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	517 # Return the first chars characters
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	518 result = self.charbuffer[:chars]
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	519 self.charbuffer = self.charbuffer[chars:]
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	520 return result
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	521
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	522 def readline(self, size=None, keepends=True):
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	523
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	524 """ Read one line from the input stream and return the
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	525 decoded data.
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	526
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	527 size, if given, is passed as size argument to the
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	528 read() method.
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	529
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	530 """
35589 5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	531 # If we have lines cached from an earlier read, return
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	532 # them unconditionally
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	533 if self.linebuffer:
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	534 line = self.linebuffer[0]
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	535 del self.linebuffer[0]
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	536 if len(self.linebuffer) == 1:
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	537 # revert to charbuffer mode; we might need more data
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	538 # next time
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	539 self.charbuffer = self.linebuffer[0]
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	540 self.linebuffer = None
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	541 if not keepends:
72485 83f43b58c988 #13012: use splitlines(keepends=True/False) instead of splitlines(0/1). Ezio Melotti <ezio.melotti@gmail.com> parents: 70437 diff changeset	542 line = line.splitlines(keepends=False)[0]
35589 5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	543 return line
36121 686306a67e9b Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 35589 diff changeset	544
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	545 readsize = size or 72
66605 7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 61862 diff changeset	546 line = self._empty_charbuffer
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	547 # If size is given, we call read() only once
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	548 while True:
35415 ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	549 data = self.read(readsize, firstline=True)
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	550 if data:
34924 c66a7fbe16cf Fix comment. Walter Dörwald <walter@livinglogic.de> parents: 34923 diff changeset	551 # If we're at a "\r" read one extra character (which might
c66a7fbe16cf Fix comment. Walter Dörwald <walter@livinglogic.de> parents: 34923 diff changeset	552 # be a "\n") to get a proper line ending. If the stream is
34923 bc123b9cef7f If the data read from the bytestream in readline() ends in a '\r' read one more Walter Dörwald <walter@livinglogic.de> parents: 34866 diff changeset	553 # temporarily exhausted we return the wrong line ending.
66605 7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 61862 diff changeset	554 if (isinstance(data, str) and data.endswith("\r")) or \
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 61862 diff changeset	555 (isinstance(data, bytes) and data.endswith(b"\r")):
34865 b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34761 diff changeset	556 data += self.read(size=1, chars=1)
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34761 diff changeset	557
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	558 line += data
72485 83f43b58c988 #13012: use splitlines(keepends=True/False) instead of splitlines(0/1). Ezio Melotti <ezio.melotti@gmail.com> parents: 70437 diff changeset	559 lines = line.splitlines(keepends=True)
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	560 if lines:
35589 5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	561 if len(lines) > 1:
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	562 # More than one line result; the first line is a full line
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	563 # to return
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	564 line = lines[0]
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	565 del lines[0]
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	566 if len(lines) > 1:
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	567 # cache the remaining lines
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	568 lines[-1] += self.charbuffer
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	569 self.linebuffer = lines
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	570 self.charbuffer = None
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	571 else:
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	572 # only one remaining line, put it back into charbuffer
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	573 self.charbuffer = lines[0] + self.charbuffer
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	574 if not keepends:
72485 83f43b58c988 #13012: use splitlines(keepends=True/False) instead of splitlines(0/1). Ezio Melotti <ezio.melotti@gmail.com> parents: 70437 diff changeset	575 line = line.splitlines(keepends=False)[0]
35589 5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	576 break
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	577 line0withend = lines[0]
72485 83f43b58c988 #13012: use splitlines(keepends=True/False) instead of splitlines(0/1). Ezio Melotti <ezio.melotti@gmail.com> parents: 70437 diff changeset	578 line0withoutend = lines[0].splitlines(keepends=False)[0]
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	579 if line0withend != line0withoutend: # We really have a line end
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	580 # Put the rest back together and keep it until the next call
66605 7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 61862 diff changeset	581 self.charbuffer = self._empty_charbuffer.join(lines[1:]) + \
7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 61862 diff changeset	582 self.charbuffer
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	583 if keepends:
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	584 line = line0withend
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	585 else:
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	586 line = line0withoutend
34387 f25dcd7f3e48 Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	587 break
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	588 # we didn't get anything or this was our only try
34387 f25dcd7f3e48 Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	589 if not data or size is not None:
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	590 if line and not keepends:
72485 83f43b58c988 #13012: use splitlines(keepends=True/False) instead of splitlines(0/1). Ezio Melotti <ezio.melotti@gmail.com> parents: 70437 diff changeset	591 line = line.splitlines(keepends=False)[0]
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	592 break
66605 7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 61862 diff changeset	593 if readsize < 8000:
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	594 readsize *= 2
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	595 return line
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	596
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	597 def readlines(self, sizehint=None, keepends=True):
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	598
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	599 """ Read all lines available on the input stream
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	600 and return them as list of lines.
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	601
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	602 Line breaks are implemented using the codec's decoder
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	603 method and are included in the list entries.
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	604
31591 26a80f0af9f3 Ignore sizehint argument. Fixes SF #844561. Marc-André Lemburg <mal@egenix.com> parents: 27618 diff changeset	605 sizehint, if given, is ignored since there is no efficient
26a80f0af9f3 Ignore sizehint argument. Fixes SF #844561. Marc-André Lemburg <mal@egenix.com> parents: 27618 diff changeset	606 way to finding the true end-of-line.
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	607
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	608 """
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	609 data = self.read()
33782 a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	610 return data.splitlines(keepends)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	611
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	612 def reset(self):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	613
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	614 """ Resets the codec buffers used for keeping state.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	615
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	616 Note that no stream repositioning should take place.
13620 5b727ab37092 Spelling fixes supplied by Rob W. W. Hooft. All these are fixes in either Thomas Wouters <thomas@python.org> parents: 12986 diff changeset	617 This method is primarily intended to be able to recover
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	618 from decoding errors.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	619
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	620 """
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	621 self.bytebuffer = b""
66605 7e4833764c88 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. Georg Brandl <georg@python.org> parents: 61862 diff changeset	622 self.charbuffer = self._empty_charbuffer
35589 5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	623 self.linebuffer = None
34751 962800ee30ef Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34707 diff changeset	624
34753 11b4350b5879 Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34751 diff changeset	625 def seek(self, offset, whence=0):
34751 962800ee30ef Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34707 diff changeset	626 """ Set the input stream's current position.
962800ee30ef Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34707 diff changeset	627
962800ee30ef Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34707 diff changeset	628 Resets the codec buffers used for keeping state.
962800ee30ef Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34707 diff changeset	629 """
61862 830cd89b01e8 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61849 diff changeset	630 self.stream.seek(offset, whence)
34751 962800ee30ef Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34707 diff changeset	631 self.reset()
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	632
41501 0c5019afc55b PEP 3114: rename .next() to .__next__() and add next() builtin. Georg Brandl <georg@python.org> parents: 41473 diff changeset	633 def __next__(self):
26082 f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	634
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	635 """ Return the next decoded line from the input stream."""
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	636 line = self.readline()
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	637 if line:
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	638 return line
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	639 raise StopIteration
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	640
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	641 def __iter__(self):
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	642 return self
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	643
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	644 def __getattr__(self, name,
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	645 getattr=getattr):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	646
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	647 """ Inherit all other methods from the underlying stream.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	648 """
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	649 return getattr(self.stream, name)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	650
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	651 def __enter__(self):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	652 return self
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	653
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	654 def __exit__(self, type, value, tb):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	655 self.stream.close()
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	656
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	657 ###
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	658
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	659 class StreamReaderWriter:
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	660
12389 e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	661 """ StreamReaderWriter instances allow wrapping streams which
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	662 work in both read and write modes.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	663
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	664 The design is such that one can use the factory functions
13620 5b727ab37092 Spelling fixes supplied by Rob W. W. Hooft. All these are fixes in either Thomas Wouters <thomas@python.org> parents: 12986 diff changeset	665 returned by the codec.lookup() function to construct the
12389 e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	666 instance.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	667
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	668 """
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	669 # Optional attributes set by the file wrappers below
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	670 encoding = 'unknown'
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	671
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	672 def __init__(self, stream, Reader, Writer, errors='strict'):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	673
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	674 """ Creates a StreamReaderWriter instance.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	675
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	676 stream must be a Stream-like object.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	677
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	678 Reader, Writer must be factory functions or classes
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	679 providing the StreamReader, StreamWriter interface resp.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	680
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	681 Error handling is done in the same way as defined for the
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	682 StreamWriter/Readers.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	683
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	684 """
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	685 self.stream = stream
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	686 self.reader = Reader(stream, errors)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	687 self.writer = Writer(stream, errors)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	688 self.errors = errors
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	689
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	690 def read(self, size=-1):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	691
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	692 return self.reader.read(size)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	693
12534 189298a1fab4 Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12389 diff changeset	694 def readline(self, size=None):
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	695
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	696 return self.reader.readline(size)
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	697
12534 189298a1fab4 Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12389 diff changeset	698 def readlines(self, sizehint=None):
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	699
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	700 return self.reader.readlines(sizehint)
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	701
41501 0c5019afc55b PEP 3114: rename .next() to .__next__() and add next() builtin. Georg Brandl <georg@python.org> parents: 41473 diff changeset	702 def __next__(self):
26082 f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	703
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	704 """ Return the next decoded line from the input stream."""
41501 0c5019afc55b PEP 3114: rename .next() to .__next__() and add next() builtin. Georg Brandl <georg@python.org> parents: 41473 diff changeset	705 return next(self.reader)
26082 f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	706
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	707 def __iter__(self):
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	708 return self
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	709
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	710 def write(self, data):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	711
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	712 return self.writer.write(data)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	713
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	714 def writelines(self, list):
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	715
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	716 return self.writer.writelines(list)
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	717
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	718 def reset(self):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	719
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	720 self.reader.reset()
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	721 self.writer.reset()
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	722
61849 1af27350f41b Merged revisions 81459 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 45835 diff changeset	723 def seek(self, offset, whence=0):
61862 830cd89b01e8 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61849 diff changeset	724 self.stream.seek(offset, whence)
830cd89b01e8 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61849 diff changeset	725 self.reader.reset()
830cd89b01e8 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61849 diff changeset	726 if whence == 0 and offset == 0:
830cd89b01e8 Merged revisions 81471-81472 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 61849 diff changeset	727 self.writer.reset()
61849 1af27350f41b Merged revisions 81459 via svnmerge from Victor Stinner <victor.stinner@haypocalc.com> parents: 45835 diff changeset	728
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	729 def __getattr__(self, name,
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	730 getattr=getattr):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	731
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	732 """ Inherit all other methods from the underlying stream.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	733 """
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	734 return getattr(self.stream, name)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	735
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	736 # these are needed to make "with codecs.open(...)" work properly
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	737
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	738 def __enter__(self):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	739 return self
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	740
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	741 def __exit__(self, type, value, tb):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	742 self.stream.close()
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	743
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	744 ###
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	745
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	746 class StreamRecoder:
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	747
12389 e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	748 """ StreamRecoder instances provide a frontend - backend
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	749 view of encoding data.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	750
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	751 They use the complete set of APIs returned by the
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	752 codecs.lookup() function to implement their task.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	753
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	754 Data written to the stream is first decoded into an
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	755 intermediate format (which is dependent on the given codec
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	756 combination) and then written to the stream using an instance
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	757 of the provided Writer class.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	758
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	759 In the other direction, data is read from the stream using a
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	760 Reader instance and then return encoded data to the caller.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	761
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	762 """
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	763 # Optional attributes set by the file wrappers below
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	764 data_encoding = 'unknown'
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	765 file_encoding = 'unknown'
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	766
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	767 def __init__(self, stream, encode, decode, Reader, Writer,
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	768 errors='strict'):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	769
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	770 """ Creates a StreamRecoder instance which implements a two-way
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	771 conversion: encode and decode work on the frontend (the
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	772 input to .read() and output of .write()) while
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	773 Reader and Writer work on the backend (reading and
12011 4fa25a292273 Oops, another in the same file; I should read the mail fully before Fred Drake <fdrake@acm.org> parents: 12010 diff changeset	774 writing to the stream).
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	775
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	776 You can use these objects to do transparent direct
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	777 recodings from e.g. latin-1 to utf-8 and back.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	778
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	779 stream must be a file-like object.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	780
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	781 encode, decode must adhere to the Codec interface, Reader,
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	782 Writer must be factory functions or classes providing the
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	783 StreamReader, StreamWriter interface resp.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	784
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	785 encode and decode are needed for the frontend translation,
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	786 Reader and Writer for the backend translation. Unicode is
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	787 used as intermediate encoding.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	788
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	789 Error handling is done in the same way as defined for the
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	790 StreamWriter/Readers.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	791
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	792 """
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	793 self.stream = stream
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	794 self.encode = encode
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	795 self.decode = decode
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	796 self.reader = Reader(stream, errors)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	797 self.writer = Writer(stream, errors)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	798 self.errors = errors
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	799
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	800 def read(self, size=-1):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	801
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	802 data = self.reader.read(size)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	803 data, bytesencoded = self.encode(data, self.errors)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	804 return data
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	805
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	806 def readline(self, size=None):
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	807
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	808 if size is None:
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	809 data = self.reader.readline()
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	810 else:
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	811 data = self.reader.readline(size)
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	812 data, bytesencoded = self.encode(data, self.errors)
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	813 return data
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	814
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	815 def readlines(self, sizehint=None):
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	816
31591 26a80f0af9f3 Ignore sizehint argument. Fixes SF #844561. Marc-André Lemburg <mal@egenix.com> parents: 27618 diff changeset	817 data = self.reader.read()
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	818 data, bytesencoded = self.encode(data, self.errors)
72485 83f43b58c988 #13012: use splitlines(keepends=True/False) instead of splitlines(0/1). Ezio Melotti <ezio.melotti@gmail.com> parents: 70437 diff changeset	819 return data.splitlines(keepends=True)
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	820
41501 0c5019afc55b PEP 3114: rename .next() to .__next__() and add next() builtin. Georg Brandl <georg@python.org> parents: 41473 diff changeset	821 def __next__(self):
26082 f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	822
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	823 """ Return the next decoded line from the input stream."""
41501 0c5019afc55b PEP 3114: rename .next() to .__next__() and add next() builtin. Georg Brandl <georg@python.org> parents: 41473 diff changeset	824 data = next(self.reader)
35508 1f2af259968b SF bug #1235646: codecs.StreamRecoder.next() now reencodes the data it reads Walter Dörwald <walter@livinglogic.de> parents: 35415 diff changeset	825 data, bytesencoded = self.encode(data, self.errors)
1f2af259968b SF bug #1235646: codecs.StreamRecoder.next() now reencodes the data it reads Walter Dörwald <walter@livinglogic.de> parents: 35415 diff changeset	826 return data
26082 f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	827
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	828 def __iter__(self):
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	829 return self
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	830
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	831 def write(self, data):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	832
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	833 data, bytesdecoded = self.decode(data, self.errors)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	834 return self.writer.write(data)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	835
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	836 def writelines(self, list):
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	837
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	838 data = ''.join(list)
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	839 data, bytesdecoded = self.decode(data, self.errors)
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	840 return self.writer.write(data)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	841
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	842 def reset(self):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	843
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	844 self.reader.reset()
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	845 self.writer.reset()
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	846
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	847 def __getattr__(self, name,
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	848 getattr=getattr):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	849
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	850 """ Inherit all other methods from the underlying stream.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	851 """
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	852 return getattr(self.stream, name)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	853
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	854 def __enter__(self):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	855 return self
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	856
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	857 def __exit__(self, type, value, tb):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	858 self.stream.close()
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	859
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	860 ### Shortcuts
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	861
70437 4d2ddd86b531 Revert my commit 3555cf6f9c98: "Issue #8796: codecs.open() calls the builtin Victor Stinner <victor.stinner@haypocalc.com> parents: 70430 diff changeset	862 def open(filename, mode='rb', encoding=None, errors='strict', buffering=1):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	863
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	864 """ Open an encoded file using the given mode and return
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	865 a wrapped version providing transparent encoding/decoding.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	866
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	867 Note: The wrapped version will only accept the object format
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	868 defined by the codecs, i.e. Unicode objects for most builtin
34761 07745166f571 typo Skip Montanaro <skip@pobox.com> parents: 34753 diff changeset	869 codecs. Output is also codec dependent and will usually be
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	870 Unicode as well.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	871
12986 5279a6e00d36 Marc-Andre Lemburg <mal@lemburg.com>: Marc-André Lemburg <mal@egenix.com> parents: 12534 diff changeset	872 Files are always opened in binary mode, even if no binary mode
27618 74d9604e9353 Fix typos. Walter Dörwald <walter@livinglogic.de> parents: 26906 diff changeset	873 was specified. This is done to avoid data loss due to encodings
12986 5279a6e00d36 Marc-Andre Lemburg <mal@lemburg.com>: Marc-André Lemburg <mal@egenix.com> parents: 12534 diff changeset	874 using 8-bit values. The default file mode is 'rb' meaning to
5279a6e00d36 Marc-Andre Lemburg <mal@lemburg.com>: Marc-André Lemburg <mal@egenix.com> parents: 12534 diff changeset	875 open the file in binary read mode.
5279a6e00d36 Marc-Andre Lemburg <mal@lemburg.com>: Marc-André Lemburg <mal@egenix.com> parents: 12534 diff changeset	876
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	877 encoding specifies the encoding which is to be used for the
27618 74d9604e9353 Fix typos. Walter Dörwald <walter@livinglogic.de> parents: 26906 diff changeset	878 file.
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	879
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	880 errors may be given to define the error handling. It defaults
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	881 to 'strict' which causes ValueErrors to be raised in case an
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	882 encoding error occurs.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	883
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	884 buffering has the same meaning as for the builtin open() API.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	885 It defaults to line buffered.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	886
12389 e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	887 The returned wrapped file object provides an extra attribute
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	888 .encoding which allows querying the used encoding. This
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	889 attribute is only available if an encoding was specified as
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	890 parameter.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	891
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	892 """
70437 4d2ddd86b531 Revert my commit 3555cf6f9c98: "Issue #8796: codecs.open() calls the builtin Victor Stinner <victor.stinner@haypocalc.com> parents: 70430 diff changeset	893 if encoding is not None and \
4d2ddd86b531 Revert my commit 3555cf6f9c98: "Issue #8796: codecs.open() calls the builtin Victor Stinner <victor.stinner@haypocalc.com> parents: 70430 diff changeset	894 'b' not in mode:
4d2ddd86b531 Revert my commit 3555cf6f9c98: "Issue #8796: codecs.open() calls the builtin Victor Stinner <victor.stinner@haypocalc.com> parents: 70430 diff changeset	895 # Force opening of the file in binary mode
4d2ddd86b531 Revert my commit 3555cf6f9c98: "Issue #8796: codecs.open() calls the builtin Victor Stinner <victor.stinner@haypocalc.com> parents: 70430 diff changeset	896 mode = mode + 'b'
4d2ddd86b531 Revert my commit 3555cf6f9c98: "Issue #8796: codecs.open() calls the builtin Victor Stinner <victor.stinner@haypocalc.com> parents: 70430 diff changeset	897 file = builtins.open(filename, mode, buffering)
4d2ddd86b531 Revert my commit 3555cf6f9c98: "Issue #8796: codecs.open() calls the builtin Victor Stinner <victor.stinner@haypocalc.com> parents: 70430 diff changeset	898 if encoding is None:
4d2ddd86b531 Revert my commit 3555cf6f9c98: "Issue #8796: codecs.open() calls the builtin Victor Stinner <victor.stinner@haypocalc.com> parents: 70430 diff changeset	899 return file
4d2ddd86b531 Revert my commit 3555cf6f9c98: "Issue #8796: codecs.open() calls the builtin Victor Stinner <victor.stinner@haypocalc.com> parents: 70430 diff changeset	900 info = lookup(encoding)
4d2ddd86b531 Revert my commit 3555cf6f9c98: "Issue #8796: codecs.open() calls the builtin Victor Stinner <victor.stinner@haypocalc.com> parents: 70430 diff changeset	901 srw = StreamReaderWriter(file, info.streamreader, info.streamwriter, errors)
4d2ddd86b531 Revert my commit 3555cf6f9c98: "Issue #8796: codecs.open() calls the builtin Victor Stinner <victor.stinner@haypocalc.com> parents: 70430 diff changeset	902 # Add attributes to simplify introspection
4d2ddd86b531 Revert my commit 3555cf6f9c98: "Issue #8796: codecs.open() calls the builtin Victor Stinner <victor.stinner@haypocalc.com> parents: 70430 diff changeset	903 srw.encoding = encoding
4d2ddd86b531 Revert my commit 3555cf6f9c98: "Issue #8796: codecs.open() calls the builtin Victor Stinner <victor.stinner@haypocalc.com> parents: 70430 diff changeset	904 return srw
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	905
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	906 def EncodedFile(file, data_encoding, file_encoding=None, errors='strict'):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	907
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	908 """ Return a wrapped version of file which provides transparent
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	909 encoding translation.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	910
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	911 Strings written to the wrapped file are interpreted according
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	912 to the given data_encoding and then written to the original
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	913 file as string using file_encoding. The intermediate encoding
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	914 will usually be Unicode but depends on the specified codecs.
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	915
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	916 Strings are read from the file using file_encoding and then
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	917 passed back to the caller as string using data_encoding.
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	918
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	919 If file_encoding is not given, it defaults to data_encoding.
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	920
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	921 errors may be given to define the error handling. It defaults
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	922 to 'strict' which causes ValueErrors to be raised in case an
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	923 encoding error occurs.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	924
12389 e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	925 The returned wrapped file object provides two extra attributes
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	926 .data_encoding and .file_encoding which reflect the given
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	927 parameters of the same name. The attributes can be used for
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	928 introspection by Python programs.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	929
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	930 """
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	931 if file_encoding is None:
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	932 file_encoding = data_encoding
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	933 data_info = lookup(data_encoding)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	934 file_info = lookup(file_encoding)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	935 sr = StreamRecoder(file, data_info.encode, data_info.decode,
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	936 file_info.streamreader, file_info.streamwriter, errors)
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	937 # Add attributes to simplify introspection
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	938 sr.data_encoding = data_encoding
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	939 sr.file_encoding = file_encoding
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	940 return sr
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	941
19981 0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	942 ### Helpers for codec lookup
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	943
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	944 def getencoder(encoding):
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	945
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	946 """ Lookup up the codec for the given encoding and return
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	947 its encoder function.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	948
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	949 Raises a LookupError in case the encoding cannot be found.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	950
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	951 """
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	952 return lookup(encoding).encode
19981 0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	953
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	954 def getdecoder(encoding):
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	955
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	956 """ Lookup up the codec for the given encoding and return
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	957 its decoder function.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	958
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	959 Raises a LookupError in case the encoding cannot be found.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	960
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	961 """
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	962 return lookup(encoding).decode
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	963
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	964 def getincrementalencoder(encoding):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	965
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	966 """ Lookup up the codec for the given encoding and return
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	967 its IncrementalEncoder class or factory function.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	968
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	969 Raises a LookupError in case the encoding cannot be found
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	970 or the codecs doesn't provide an incremental encoder.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	971
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	972 """
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	973 encoder = lookup(encoding).incrementalencoder
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	974 if encoder is None:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	975 raise LookupError(encoding)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	976 return encoder
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	977
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	978 def getincrementaldecoder(encoding):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	979
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	980 """ Lookup up the codec for the given encoding and return
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	981 its IncrementalDecoder class or factory function.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	982
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	983 Raises a LookupError in case the encoding cannot be found
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	984 or the codecs doesn't provide an incremental decoder.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	985
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	986 """
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	987 decoder = lookup(encoding).incrementaldecoder
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	988 if decoder is None:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	989 raise LookupError(encoding)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	990 return decoder
19981 0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	991
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	992 def getreader(encoding):
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	993
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	994 """ Lookup up the codec for the given encoding and return
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	995 its StreamReader class or factory function.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	996
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	997 Raises a LookupError in case the encoding cannot be found.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	998
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	999 """
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1000 return lookup(encoding).streamreader
19981 0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	1001
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	1002 def getwriter(encoding):
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	1003
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	1004 """ Lookup up the codec for the given encoding and return
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	1005 its StreamWriter class or factory function.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	1006
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	1007 Raises a LookupError in case the encoding cannot be found.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	1008
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	1009 """
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1010 return lookup(encoding).streamwriter
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1011
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1012 def iterencode(iterator, encoding, errors='strict', **kwargs):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1013 """
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1014 Encoding iterator.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1015
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1016 Encodes the input strings from the iterator using a IncrementalEncoder.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1017
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1018 errors and kwargs are passed through to the IncrementalEncoder
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1019 constructor.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1020 """
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1021 encoder = getincrementalencoder(encoding)(errors, **kwargs)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1022 for input in iterator:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1023 output = encoder.encode(input)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1024 if output:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1025 yield output
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1026 output = encoder.encode("", True)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1027 if output:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1028 yield output
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1029
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1030 def iterdecode(iterator, encoding, errors='strict', **kwargs):
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1031 """
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1032 Decoding iterator.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1033
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1034 Decodes the input strings from the iterator using a IncrementalDecoder.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1035
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1036 errors and kwargs are passed through to the IncrementalDecoder
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1037 constructor.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1038 """
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1039 decoder = getincrementaldecoder(encoding)(errors, **kwargs)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1040 for input in iterator:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1041 output = decoder.decode(input)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1042 if output:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1043 yield output
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1044 output = decoder.decode(b"", True)
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1045 if output:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36855 diff changeset	1046 yield output
19981 0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	1047
15706 bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	1048 ### Helpers for charmap-based codecs
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	1049
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	1050 def make_identity_dict(rng):
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	1051
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	1052 """ make_identity_dict(rng) -> dict
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	1053
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	1054 Return a dictionary where elements of the rng sequence are
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	1055 mapped to themselves.
15867 f5a3d98a35bc Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 15706 diff changeset	1056
15706 bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	1057 """
77471 8f3a5308f50b Issue #14874: Restore charmap decoding speed to pre-PEP 393 levels. Antoine Pitrou <solipsis@pitrou.net> parents: 72485 diff changeset	1058 return {i:i for i in rng}
15706 bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	1059
17843 3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1060 def make_encoding_map(decoding_map):
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1061
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1062 """ Creates an encoding map from a decoding map.
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1063
27618 74d9604e9353 Fix typos. Walter Dörwald <walter@livinglogic.de> parents: 26906 diff changeset	1064 If a target mapping in the decoding map occurs multiple
17843 3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1065 times, then that target is mapped to None (undefined mapping),
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1066 causing an exception when encountered by the charmap codec
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1067 during translation.
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1068
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1069 One example where this happens is cp875.py which decodes
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1070 multiple character to \u001a.
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1071
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1072 """
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1073 m = {}
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1074 for k,v in decoding_map.items():
23620 a1de0bab5475 SF 563203. Replaced 'has_key()' with 'in'. Raymond Hettinger <python@rcn.com> parents: 22149 diff changeset	1075 if not v in m:
17843 3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1076 m[v] = k
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1077 else:
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1078 m[v] = None
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1079 return m
17978 5e87a08abb05 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 17843 diff changeset	1080
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	1081 ### error handlers
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	1082
34707 0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	1083 try:
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	1084 strict_errors = lookup_error("strict")
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	1085 ignore_errors = lookup_error("ignore")
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	1086 replace_errors = lookup_error("replace")
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	1087 xmlcharrefreplace_errors = lookup_error("xmlcharrefreplace")
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	1088 backslashreplace_errors = lookup_error("backslashreplace")
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	1089 except LookupError:
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	1090 # In --disable-unicode builds, these error handler are missing
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	1091 strict_errors = None
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	1092 ignore_errors = None
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	1093 replace_errors = None
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	1094 xmlcharrefreplace_errors = None
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	1095 backslashreplace_errors = None
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	1096
18904 5af1ee422575 Add dead imports of modules that are "magically" imported. Martin v. Löwis <martin@v.loewis.de> parents: 17978 diff changeset	1097 # Tell modulefinder that using codecs probably needs the encodings
5af1ee422575 Add dead imports of modules that are "magically" imported. Martin v. Löwis <martin@v.loewis.de> parents: 17978 diff changeset	1098 # package
5af1ee422575 Add dead imports of modules that are "magically" imported. Martin v. Löwis <martin@v.loewis.de> parents: 17978 diff changeset	1099 _false = 0
5af1ee422575 Add dead imports of modules that are "magically" imported. Martin v. Löwis <martin@v.loewis.de> parents: 17978 diff changeset	1100 if _false:
5af1ee422575 Add dead imports of modules that are "magically" imported. Martin v. Löwis <martin@v.loewis.de> parents: 17978 diff changeset	1101 import encodings
5af1ee422575 Add dead imports of modules that are "magically" imported. Martin v. Löwis <martin@v.loewis.de> parents: 17978 diff changeset	1102
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	1103 ### Tests
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	1104
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	1105 if __name__ == '__main__':
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	1106
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	1107 # Make stdout translate Latin-1 output into UTF-8 output
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	1108 sys.stdout = EncodedFile(sys.stdout, 'latin-1', 'utf-8')
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	1109
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	1110 # Have stdin translate Latin-1 input into UTF-8 input
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	1111 sys.stdin = EncodedFile(sys.stdin, 'utf-8', 'latin-1')

Mercurial > cpython-lru_cache-weakref

annotate Lib/codecs.py @ 87690:66c1c9f32567 default tip