cpython-lru_cache-weakref: Lib/codecs.py annotate

annotate Lib/codecs.py @ 48715:48dddfd2fbf3

RPC_WSTR is not available Visual Studio 2003 or earlier. (Maybe I should have defined RPC_WSTR on old compiler, but uuidcreate() is only place using it, so I simply replaced with unsigned short *)

author	Hirokazu Yamamoto <ocean-city@m2.ccsnet.ne.jp>
date	Sun, 17 Aug 2008 07:26:26 +0000
parents	04050373d799
children	1af27350f41b

rev	line source
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	1 """ codecs -- Python Codec Registry, API and helpers.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	2
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	3
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	4 Written by Marc-Andre Lemburg (mal@lemburg.com).
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	5
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	6 (c) Copyright CNRI, All Rights Reserved. NO WARRANTY.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	7
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	8 """#"
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	9
44098 0889daf6d0d9 #1535: rename __builtin__ module to builtins. Georg Brandl <georg@python.org> parents: 41606 diff changeset	10 import builtins, sys
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	11
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	12 ### Registry and builtin stateless codec functions
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	13
12195 bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12051 diff changeset	14 try:
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12051 diff changeset	15 from _codecs import *
40587 6b18a095c2e7 SF patch 1631942 by Collin Winter: Guido van Rossum <guido@python.org> parents: 40450 diff changeset	16 except ImportError as why:
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	17 raise SystemError('Failed to load the builtin codecs: %s' % why)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	18
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	19 __all__ = ["register", "lookup", "open", "EncodedFile", "BOM", "BOM_BE",
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	20 "BOM_LE", "BOM32_BE", "BOM32_LE", "BOM64_BE", "BOM64_LE",
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	21 "BOM_UTF8", "BOM_UTF16", "BOM_UTF16_LE", "BOM_UTF16_BE",
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	22 "BOM_UTF32", "BOM_UTF32_LE", "BOM_UTF32_BE",
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	23 "strict_errors", "ignore_errors", "replace_errors",
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	24 "xmlcharrefreplace_errors",
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	25 "register_error", "lookup_error"]
16070 b5345e99596b added __all__ lists to a number of Python modules Skip Montanaro <skip@pobox.com> parents: 15867 diff changeset	26
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	27 ### Constants
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	28
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	29 #
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	30 # Byte Order Mark (BOM = ZERO WIDTH NO-BREAK SPACE = U+FEFF)
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	31 # and its possible byte string values
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	32 # for UTF8/UTF16/UTF32 output and little/big endian machines
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	33 #
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	34
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	35 # UTF-8
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	36 BOM_UTF8 = b'\xef\xbb\xbf'
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	37
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	38 # UTF-16, little endian
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	39 BOM_LE = BOM_UTF16_LE = b'\xff\xfe'
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	40
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	41 # UTF-16, big endian
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	42 BOM_BE = BOM_UTF16_BE = b'\xfe\xff'
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	43
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	44 # UTF-32, little endian
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	45 BOM_UTF32_LE = b'\xff\xfe\x00\x00'
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	46
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	47 # UTF-32, big endian
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	48 BOM_UTF32_BE = b'\x00\x00\xfe\xff'
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	49
26616 20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	50 if sys.byteorder == 'little':
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	51
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	52 # UTF-16, native endianness
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	53 BOM = BOM_UTF16 = BOM_UTF16_LE
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	54
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	55 # UTF-32, native endianness
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	56 BOM_UTF32 = BOM_UTF32_LE
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	57
26616 20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	58 else:
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	59
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	60 # UTF-16, native endianness
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	61 BOM = BOM_UTF16 = BOM_UTF16_BE
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	62
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	63 # UTF-32, native endianness
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	64 BOM_UTF32 = BOM_UTF32_BE
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	65
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	66 # Old broken names (don't use in new code)
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	67 BOM32_LE = BOM_UTF16_LE
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	68 BOM32_BE = BOM_UTF16_BE
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	69 BOM64_LE = BOM_UTF32_LE
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	70 BOM64_BE = BOM_UTF32_BE
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	71
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	72
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	73 ### Codec base classes (defining the API)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	74
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	75 class CodecInfo(tuple):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	76
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	77 def __new__(cls, encode, decode, streamreader=None, streamwriter=None,
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	78 incrementalencoder=None, incrementaldecoder=None, name=None):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	79 self = tuple.__new__(cls, (encode, decode, streamreader, streamwriter))
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	80 self.name = name
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	81 self.encode = encode
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	82 self.decode = decode
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	83 self.incrementalencoder = incrementalencoder
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	84 self.incrementaldecoder = incrementaldecoder
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	85 self.streamwriter = streamwriter
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	86 self.streamreader = streamreader
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	87 return self
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	88
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	89 def __repr__(self):
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	90 return "<%s.%s object for encoding %s at 0x%x>" % \
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	91 (self.__class__.__module__, self.__class__.__name__,
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	92 self.name, id(self))
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	93
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	94 class Codec:
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	95
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	96 """ Defines the interface for stateless encoders/decoders.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	97
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	98 The .encode()/.decode() methods may use different error
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	99 handling schemes by providing the errors argument. These
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	100 string values are predefined:
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	101
12051 d89556c04ef9 Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12011 diff changeset	102 'strict' - raise a ValueError error (or a subclass)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	103 'ignore' - ignore the character and continue with the next
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	104 'replace' - replace with a suitable replacement character;
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	105 Python will use the official U+FFFD REPLACEMENT
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	106 CHARACTER for the builtin Unicode codecs on
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	107 decoding and '?' on encoding.
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	108 'xmlcharrefreplace' - Replace with the appropriate XML
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	109 character reference (only for encoding).
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	110 'backslashreplace' - Replace with backslashed escape sequences
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	111 (only for encoding).
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	112
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	113 The set of allowed values can be extended via register_error.
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	114
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	115 """
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	116 def encode(self, input, errors='strict'):
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	117
12010 bd1014215f62 "intput" --> "input" (in docstring); noted by Detlef Lannert Fred Drake <fdrake@acm.org> parents: 11990 diff changeset	118 """ Encodes the object input and returns a tuple (output
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	119 object, length consumed).
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	120
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	121 errors defines the error handling to apply. It defaults to
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	122 'strict' handling.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	123
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	124 The method may not store state in the Codec instance. Use
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	125 StreamCodec for codecs which have to keep state in order to
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	126 make encoding/decoding efficient.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	127
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	128 The encoder must be able to handle zero length input and
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	129 return an empty object of the output object type in this
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	130 situation.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	131
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	132 """
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	133 raise NotImplementedError
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	134
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	135 def decode(self, input, errors='strict'):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	136
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	137 """ Decodes the object input and returns a tuple (output
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	138 object, length consumed).
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	139
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	140 input must be an object which provides the bf_getreadbuf
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	141 buffer slot. Python strings, buffer objects and memory
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	142 mapped files are examples of objects providing this slot.
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	143
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	144 errors defines the error handling to apply. It defaults to
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	145 'strict' handling.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	146
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	147 The method may not store state in the Codec instance. Use
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	148 StreamCodec for codecs which have to keep state in order to
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	149 make encoding/decoding efficient.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	150
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	151 The decoder must be able to handle zero length input and
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	152 return an empty object of the output object type in this
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	153 situation.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	154
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	155 """
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	156 raise NotImplementedError
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	157
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	158 class IncrementalEncoder(object):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	159 """
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	160 An IncrementalEncoder encodes an input in multiple steps. The input can
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	161 be passed piece by piece to the encode() method. The IncrementalEncoder
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	162 remembers the state of the encoding process between calls to encode().
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	163 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	164 def __init__(self, errors='strict'):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	165 """
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	166 Creates an IncrementalEncoder instance.
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	167
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	168 The IncrementalEncoder may use different error handling schemes by
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	169 providing the errors keyword argument. See the module docstring
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	170 for a list of possible values.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	171 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	172 self.errors = errors
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	173 self.buffer = ""
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	174
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	175 def encode(self, input, final=False):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	176 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	177 Encodes input and returns the resulting object.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	178 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	179 raise NotImplementedError
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	180
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	181 def reset(self):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	182 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	183 Resets the encoder to the initial state.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	184 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	185
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	186 def getstate(self):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	187 """
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	188 Return the current state of the encoder.
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	189 """
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	190 return 0
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	191
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	192 def setstate(self, state):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	193 """
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	194 Set the current state of the encoder. state must have been
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	195 returned by getstate().
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	196 """
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	197
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	198 class BufferedIncrementalEncoder(IncrementalEncoder):
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	199 """
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	200 This subclass of IncrementalEncoder can be used as the baseclass for an
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	201 incremental encoder if the encoder must keep some of the output in a
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	202 buffer between calls to encode().
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	203 """
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	204 def __init__(self, errors='strict'):
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	205 IncrementalEncoder.__init__(self, errors)
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	206 # unencoded input that is kept between calls to encode()
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	207 self.buffer = ""
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	208
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	209 def _buffer_encode(self, input, errors, final):
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	210 # Overwrite this method in subclasses: It must encode input
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	211 # and return an (output, length consumed) tuple
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	212 raise NotImplementedError
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	213
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	214 def encode(self, input, final=False):
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	215 # encode input (taking the buffer into account)
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	216 data = self.buffer + input
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	217 (result, consumed) = self._buffer_encode(data, self.errors, final)
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	218 # keep unencoded input until the next call
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	219 self.buffer = data[consumed:]
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	220 return result
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	221
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	222 def reset(self):
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	223 IncrementalEncoder.reset(self)
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	224 self.buffer = ""
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 37875 diff changeset	225
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	226 def getstate(self):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	227 return self.buffer or 0
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	228
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	229 def setstate(self, state):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	230 self.buffer = state or ""
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	231
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	232 class IncrementalDecoder(object):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	233 """
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	234 An IncrementalDecoder decodes an input in multiple steps. The input can
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	235 be passed piece by piece to the decode() method. The IncrementalDecoder
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	236 remembers the state of the decoding process between calls to decode().
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	237 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	238 def __init__(self, errors='strict'):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	239 """
45835 04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	240 Create a IncrementalDecoder instance.
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	241
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	242 The IncrementalDecoder may use different error handling schemes by
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	243 providing the errors keyword argument. See the module docstring
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	244 for a list of possible values.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	245 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	246 self.errors = errors
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	247
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	248 def decode(self, input, final=False):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	249 """
45835 04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	250 Decode input and returns the resulting object.
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	251 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	252 raise NotImplementedError
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	253
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	254 def reset(self):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	255 """
45835 04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	256 Reset the decoder to the initial state.
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	257 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	258
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	259 def getstate(self):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	260 """
45835 04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	261 Return the current state of the decoder.
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	262
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	263 This must be a (buffered_input, additional_state_info) tuple.
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	264 buffered_input must be a bytes object containing bytes that
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	265 were passed to decode() that have not yet been converted.
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	266 additional_state_info must be a non-negative integer
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	267 representing the state of the decoder WITHOUT yet having
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	268 processed the contents of buffered_input. In the initial state
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	269 and after reset(), getstate() must return (b"", 0).
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	270 """
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	271 return (b"", 0)
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	272
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	273 def setstate(self, state):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	274 """
45835 04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	275 Set the current state of the decoder.
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	276
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	277 state must have been returned by getstate(). The effect of
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders; Ka-Ping Yee <ping@zesty.ca> parents: 45806 diff changeset	278 setstate((b"", 0)) must be equivalent to reset().
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	279 """
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	280
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	281 class BufferedIncrementalDecoder(IncrementalDecoder):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	282 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	283 This subclass of IncrementalDecoder can be used as the baseclass for an
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	284 incremental decoder if the decoder must be able to handle incomplete
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	285 byte sequences.
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	286 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	287 def __init__(self, errors='strict'):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	288 IncrementalDecoder.__init__(self, errors)
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	289 # undecoded input that is kept between calls to decode()
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	290 self.buffer = b""
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	291
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	292 def _buffer_decode(self, input, errors, final):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	293 # Overwrite this method in subclasses: It must decode input
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	294 # and return an (output, length consumed) tuple
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	295 raise NotImplementedError
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	296
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	297 def decode(self, input, final=False):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	298 # decode input (taking the buffer into account)
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	299 data = self.buffer + input
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	300 (result, consumed) = self._buffer_decode(data, self.errors, final)
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	301 # keep undecoded input until the next call
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	302 self.buffer = data[consumed:]
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	303 return result
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	304
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	305 def reset(self):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	306 IncrementalDecoder.reset(self)
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	307 self.buffer = b""
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	308
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	309 def getstate(self):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	310 # additional state info is always 0
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	311 return (self.buffer, 0)
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	312
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	313 def setstate(self, state):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	314 # ignore additional state info
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	315 self.buffer = state[0]
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	316
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	317 #
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	318 # The StreamWriter and StreamReader class provide generic working
19967 0b51e607f333 Fix typo in comment Andrew M. Kuchling <amk@amk.ca> parents: 18936 diff changeset	319 # interfaces which can be used to implement new encoding submodules
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	320 # very easily. See encodings/utf_8.py for an example on how this is
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	321 # done.
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	322 #
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	323
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	324 class StreamWriter(Codec):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	325
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	326 def __init__(self, stream, errors='strict'):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	327
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	328 """ Creates a StreamWriter instance.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	329
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	330 stream must be a file-like object open for writing
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	331 (binary) data.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	332
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	333 The StreamWriter may use different error handling
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	334 schemes by providing the errors keyword argument. These
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	335 parameters are predefined:
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	336
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	337 'strict' - raise a ValueError (or a subclass)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	338 'ignore' - ignore the character and continue with the next
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	339 'replace'- replace with a suitable replacement character
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	340 'xmlcharrefreplace' - Replace with the appropriate XML
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	341 character reference.
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	342 'backslashreplace' - Replace with backslashed escape
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	343 sequences (only for encoding).
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	344
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	345 The set of allowed parameter values can be extended via
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	346 register_error.
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	347 """
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	348 self.stream = stream
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	349 self.errors = errors
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	350
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	351 def write(self, object):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	352
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	353 """ Writes the object's contents encoded to self.stream.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	354 """
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	355 data, consumed = self.encode(object, self.errors)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	356 self.stream.write(data)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	357
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	358 def writelines(self, list):
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	359
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	360 """ Writes the concatenated list of strings to the stream
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	361 using .write().
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	362 """
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	363 self.write(''.join(list))
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	364
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	365 def reset(self):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	366
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	367 """ Flushes and resets the codec buffers used for keeping state.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	368
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	369 Calling this method should ensure that the data on the
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	370 output is put into a clean state, that allows appending
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	371 of new fresh data without having to rescan the whole
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	372 stream to recover state.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	373
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	374 """
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	375 pass
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	376
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	377 def __getattr__(self, name,
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	378 getattr=getattr):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	379
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	380 """ Inherit all other methods from the underlying stream.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	381 """
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	382 return getattr(self.stream, name)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	383
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	384 def __enter__(self):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	385 return self
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	386
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	387 def __exit__(self, type, value, tb):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	388 self.stream.close()
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	389
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	390 ###
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	391
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	392 class StreamReader(Codec):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	393
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	394 def __init__(self, stream, errors='strict'):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	395
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	396 """ Creates a StreamReader instance.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	397
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	398 stream must be a file-like object open for reading
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	399 (binary) data.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	400
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	401 The StreamReader may use different error handling
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	402 schemes by providing the errors keyword argument. These
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	403 parameters are predefined:
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	404
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	405 'strict' - raise a ValueError (or a subclass)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	406 'ignore' - ignore the character and continue with the next
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	407 'replace'- replace with a suitable replacement character;
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	408
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	409 The set of allowed parameter values can be extended via
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	410 register_error.
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	411 """
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	412 self.stream = stream
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	413 self.errors = errors
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	414 self.bytebuffer = b""
35269 2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34924 diff changeset	415 # For str->str decoding this will stay a str
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34924 diff changeset	416 # For str->unicode decoding the first read will promote it to unicode
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34924 diff changeset	417 self.charbuffer = ""
35589 5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	418 self.linebuffer = None
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	419
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	420 def decode(self, input, errors='strict'):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	421 raise NotImplementedError
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	422
35415 ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	423 def read(self, size=-1, chars=-1, firstline=False):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	424
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	425 """ Decodes data from the stream self.stream and returns the
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	426 resulting object.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	427
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	428 chars indicates the number of characters to read from the
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	429 stream. read() will never return more than chars
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	430 characters, but it might return less, if there are not enough
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	431 characters available.
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	432
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	433 size indicates the approximate maximum number of bytes to
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	434 read from the stream for decoding purposes. The decoder
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	435 can modify this setting as appropriate. The default value
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	436 -1 indicates to read and decode as much as possible. size
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	437 is intended to prevent having to decode huge files in one
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	438 step.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	439
35415 ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	440 If firstline is true, and a UnicodeDecodeError happens
ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	441 after the first line terminator in the input only the first line
ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	442 will be returned, the rest of the input will be kept until the
ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	443 next call to read().
ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	444
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	445 The method should use a greedy read strategy meaning that
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	446 it should read as much data as is allowed within the
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	447 definition of the encoding and the given size, e.g. if
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	448 optional encoding endings or state markers are available
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	449 on the stream, these should be read too.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	450 """
35589 5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	451 # If we have lines cached, first merge them back into characters
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	452 if self.linebuffer:
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	453 self.charbuffer = "".join(self.linebuffer)
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	454 self.linebuffer = None
36121 686306a67e9b Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 35589 diff changeset	455
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	456 # read until we get the required number of characters (if available)
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	457 while True:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	458 # can the request can be satisfied from the character buffer?
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	459 if chars < 0:
36855 204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	460 if size < 0:
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	461 if self.charbuffer:
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	462 break
204c307b01ce If size is specified, try to read at least size characters. Walter Dörwald <walter@livinglogic.de> parents: 36121 diff changeset	463 elif len(self.charbuffer) >= size:
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	464 break
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	465 else:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	466 if len(self.charbuffer) >= chars:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	467 break
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	468 # we need more data
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	469 if size < 0:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	470 newdata = self.stream.read()
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	471 else:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	472 newdata = self.stream.read(size)
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	473 # decode bytes (those remaining from the last call included)
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	474 data = self.bytebuffer + newdata
35415 ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	475 try:
ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	476 newchars, decodedbytes = self.decode(data, self.errors)
40587 6b18a095c2e7 SF patch 1631942 by Collin Winter: Guido van Rossum <guido@python.org> parents: 40450 diff changeset	477 except UnicodeDecodeError as exc:
35415 ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	478 if firstline:
41473 a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	479 newchars, decodedbytes = \
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate() Walter Dörwald <walter@livinglogic.de> parents: 40587 diff changeset	480 self.decode(data[:exc.start], self.errors)
35415 ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	481 lines = newchars.splitlines(True)
ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	482 if len(lines)<=1:
ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	483 raise
ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	484 else:
ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	485 raise
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	486 # keep undecoded bytes until the next call
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	487 self.bytebuffer = data[decodedbytes:]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	488 # put new characters in the character buffer
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	489 self.charbuffer += newchars
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	490 # there was no data available
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	491 if not newdata:
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	492 break
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	493 if chars < 0:
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	494 # Return everything we've got
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	495 result = self.charbuffer
35269 2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34924 diff changeset	496 self.charbuffer = ""
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	497 else:
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	498 # Return the first chars characters
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	499 result = self.charbuffer[:chars]
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	500 self.charbuffer = self.charbuffer[chars:]
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	501 return result
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	502
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	503 def readline(self, size=None, keepends=True):
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	504
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	505 """ Read one line from the input stream and return the
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	506 decoded data.
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	507
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	508 size, if given, is passed as size argument to the
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	509 read() method.
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	510
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	511 """
35589 5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	512 # If we have lines cached from an earlier read, return
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	513 # them unconditionally
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	514 if self.linebuffer:
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	515 line = self.linebuffer[0]
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	516 del self.linebuffer[0]
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	517 if len(self.linebuffer) == 1:
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	518 # revert to charbuffer mode; we might need more data
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	519 # next time
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	520 self.charbuffer = self.linebuffer[0]
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	521 self.linebuffer = None
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	522 if not keepends:
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	523 line = line.splitlines(False)[0]
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	524 return line
36121 686306a67e9b Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 35589 diff changeset	525
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	526 readsize = size or 72
35269 2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34924 diff changeset	527 line = ""
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	528 # If size is given, we call read() only once
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	529 while True:
35415 ff2e66fc0a29 Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35269 diff changeset	530 data = self.read(readsize, firstline=True)
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	531 if data:
34924 c66a7fbe16cf Fix comment. Walter Dörwald <walter@livinglogic.de> parents: 34923 diff changeset	532 # If we're at a "\r" read one extra character (which might
c66a7fbe16cf Fix comment. Walter Dörwald <walter@livinglogic.de> parents: 34923 diff changeset	533 # be a "\n") to get a proper line ending. If the stream is
34923 bc123b9cef7f If the data read from the bytestream in readline() ends in a '\r' read one more Walter Dörwald <walter@livinglogic.de> parents: 34866 diff changeset	534 # temporarily exhausted we return the wrong line ending.
35269 2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34924 diff changeset	535 if data.endswith("\r"):
34865 b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34761 diff changeset	536 data += self.read(size=1, chars=1)
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if Walter Dörwald <walter@livinglogic.de> parents: 34761 diff changeset	537
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	538 line += data
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	539 lines = line.splitlines(True)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	540 if lines:
35589 5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	541 if len(lines) > 1:
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	542 # More than one line result; the first line is a full line
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	543 # to return
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	544 line = lines[0]
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	545 del lines[0]
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	546 if len(lines) > 1:
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	547 # cache the remaining lines
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	548 lines[-1] += self.charbuffer
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	549 self.linebuffer = lines
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	550 self.charbuffer = None
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	551 else:
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	552 # only one remaining line, put it back into charbuffer
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	553 self.charbuffer = lines[0] + self.charbuffer
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	554 if not keepends:
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	555 line = line.splitlines(False)[0]
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	556 break
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	557 line0withend = lines[0]
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	558 line0withoutend = lines[0].splitlines(False)[0]
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	559 if line0withend != line0withoutend: # We really have a line end
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	560 # Put the rest back together and keep it until the next call
35269 2b117eb18944 Make attributes and local variables in the StreamReader str objects instead Walter Dörwald <walter@livinglogic.de> parents: 34924 diff changeset	561 self.charbuffer = "".join(lines[1:]) + self.charbuffer
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	562 if keepends:
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	563 line = line0withend
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	564 else:
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	565 line = line0withoutend
34387 f25dcd7f3e48 Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	566 break
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	567 # we didn't get anything or this was our only try
34387 f25dcd7f3e48 Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	568 if not data or size is not None:
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	569 if line and not keepends:
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	570 line = line.splitlines(False)[0]
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	571 break
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	572 if readsize<8000:
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	573 readsize *= 2
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	574 return line
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	575
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	576 def readlines(self, sizehint=None, keepends=True):
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	577
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	578 """ Read all lines available on the input stream
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	579 and return them as list of lines.
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	580
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	581 Line breaks are implemented using the codec's decoder
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	582 method and are included in the list entries.
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	583
31591 26a80f0af9f3 Ignore sizehint argument. Fixes SF #844561. Marc-André Lemburg <mal@egenix.com> parents: 27618 diff changeset	584 sizehint, if given, is ignored since there is no efficient
26a80f0af9f3 Ignore sizehint argument. Fixes SF #844561. Marc-André Lemburg <mal@egenix.com> parents: 27618 diff changeset	585 way to finding the true end-of-line.
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	586
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	587 """
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	588 data = self.read()
33782 a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	589 return data.splitlines(keepends)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	590
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	591 def reset(self):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	592
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	593 """ Resets the codec buffers used for keeping state.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	594
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	595 Note that no stream repositioning should take place.
13620 5b727ab37092 Spelling fixes supplied by Rob W. W. Hooft. All these are fixes in either Thomas Wouters <thomas@python.org> parents: 12986 diff changeset	596 This method is primarily intended to be able to recover
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	597 from decoding errors.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	598
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	599 """
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	600 self.bytebuffer = b""
41575 beacf40b5960 Rip out all the u"..." literals and calls to unicode(). Guido van Rossum <guido@python.org> parents: 41501 diff changeset	601 self.charbuffer = ""
35589 5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35508 diff changeset	602 self.linebuffer = None
34751 962800ee30ef Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34707 diff changeset	603
34753 11b4350b5879 Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34751 diff changeset	604 def seek(self, offset, whence=0):
34751 962800ee30ef Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34707 diff changeset	605 """ Set the input stream's current position.
962800ee30ef Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34707 diff changeset	606
962800ee30ef Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34707 diff changeset	607 Resets the codec buffers used for keeping state.
962800ee30ef Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34707 diff changeset	608 """
962800ee30ef Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34707 diff changeset	609 self.reset()
962800ee30ef Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34707 diff changeset	610 self.stream.seek(offset, whence)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	611
41501 0c5019afc55b PEP 3114: rename .next() to .__next__() and add next() builtin. Georg Brandl <georg@python.org> parents: 41473 diff changeset	612 def __next__(self):
26082 f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	613
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	614 """ Return the next decoded line from the input stream."""
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	615 line = self.readline()
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	616 if line:
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	617 return line
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	618 raise StopIteration
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	619
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	620 def __iter__(self):
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	621 return self
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	622
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	623 def __getattr__(self, name,
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	624 getattr=getattr):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	625
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	626 """ Inherit all other methods from the underlying stream.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	627 """
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	628 return getattr(self.stream, name)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	629
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	630 def __enter__(self):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	631 return self
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	632
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	633 def __exit__(self, type, value, tb):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	634 self.stream.close()
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	635
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	636 ###
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	637
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	638 class StreamReaderWriter:
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	639
12389 e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	640 """ StreamReaderWriter instances allow wrapping streams which
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	641 work in both read and write modes.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	642
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	643 The design is such that one can use the factory functions
13620 5b727ab37092 Spelling fixes supplied by Rob W. W. Hooft. All these are fixes in either Thomas Wouters <thomas@python.org> parents: 12986 diff changeset	644 returned by the codec.lookup() function to construct the
12389 e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	645 instance.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	646
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	647 """
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	648 # Optional attributes set by the file wrappers below
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	649 encoding = 'unknown'
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	650
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	651 def __init__(self, stream, Reader, Writer, errors='strict'):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	652
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	653 """ Creates a StreamReaderWriter instance.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	654
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	655 stream must be a Stream-like object.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	656
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	657 Reader, Writer must be factory functions or classes
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	658 providing the StreamReader, StreamWriter interface resp.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	659
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	660 Error handling is done in the same way as defined for the
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	661 StreamWriter/Readers.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	662
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	663 """
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	664 self.stream = stream
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	665 self.reader = Reader(stream, errors)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	666 self.writer = Writer(stream, errors)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	667 self.errors = errors
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	668
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	669 def read(self, size=-1):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	670
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	671 return self.reader.read(size)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	672
12534 189298a1fab4 Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12389 diff changeset	673 def readline(self, size=None):
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	674
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	675 return self.reader.readline(size)
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	676
12534 189298a1fab4 Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12389 diff changeset	677 def readlines(self, sizehint=None):
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	678
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	679 return self.reader.readlines(sizehint)
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	680
41501 0c5019afc55b PEP 3114: rename .next() to .__next__() and add next() builtin. Georg Brandl <georg@python.org> parents: 41473 diff changeset	681 def __next__(self):
26082 f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	682
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	683 """ Return the next decoded line from the input stream."""
41501 0c5019afc55b PEP 3114: rename .next() to .__next__() and add next() builtin. Georg Brandl <georg@python.org> parents: 41473 diff changeset	684 return next(self.reader)
26082 f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	685
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	686 def __iter__(self):
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	687 return self
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	688
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	689 def write(self, data):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	690
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	691 return self.writer.write(data)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	692
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	693 def writelines(self, list):
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	694
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	695 return self.writer.writelines(list)
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	696
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	697 def reset(self):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	698
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	699 self.reader.reset()
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	700 self.writer.reset()
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	701
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	702 def __getattr__(self, name,
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	703 getattr=getattr):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	704
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	705 """ Inherit all other methods from the underlying stream.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	706 """
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	707 return getattr(self.stream, name)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	708
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	709 # these are needed to make "with codecs.open(...)" work properly
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	710
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	711 def __enter__(self):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	712 return self
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	713
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	714 def __exit__(self, type, value, tb):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	715 self.stream.close()
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	716
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	717 ###
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	718
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	719 class StreamRecoder:
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	720
12389 e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	721 """ StreamRecoder instances provide a frontend - backend
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	722 view of encoding data.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	723
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	724 They use the complete set of APIs returned by the
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	725 codecs.lookup() function to implement their task.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	726
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	727 Data written to the stream is first decoded into an
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	728 intermediate format (which is dependent on the given codec
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	729 combination) and then written to the stream using an instance
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	730 of the provided Writer class.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	731
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	732 In the other direction, data is read from the stream using a
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	733 Reader instance and then return encoded data to the caller.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	734
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	735 """
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	736 # Optional attributes set by the file wrappers below
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	737 data_encoding = 'unknown'
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	738 file_encoding = 'unknown'
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	739
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	740 def __init__(self, stream, encode, decode, Reader, Writer,
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	741 errors='strict'):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	742
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	743 """ Creates a StreamRecoder instance which implements a two-way
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	744 conversion: encode and decode work on the frontend (the
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	745 input to .read() and output of .write()) while
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	746 Reader and Writer work on the backend (reading and
12011 4fa25a292273 Oops, another in the same file; I should read the mail fully before Fred Drake <fdrake@acm.org> parents: 12010 diff changeset	747 writing to the stream).
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	748
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	749 You can use these objects to do transparent direct
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	750 recodings from e.g. latin-1 to utf-8 and back.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	751
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	752 stream must be a file-like object.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	753
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	754 encode, decode must adhere to the Codec interface, Reader,
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	755 Writer must be factory functions or classes providing the
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	756 StreamReader, StreamWriter interface resp.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	757
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	758 encode and decode are needed for the frontend translation,
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	759 Reader and Writer for the backend translation. Unicode is
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	760 used as intermediate encoding.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	761
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	762 Error handling is done in the same way as defined for the
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	763 StreamWriter/Readers.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	764
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	765 """
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	766 self.stream = stream
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	767 self.encode = encode
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	768 self.decode = decode
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	769 self.reader = Reader(stream, errors)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	770 self.writer = Writer(stream, errors)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	771 self.errors = errors
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	772
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	773 def read(self, size=-1):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	774
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	775 data = self.reader.read(size)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	776 data, bytesencoded = self.encode(data, self.errors)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	777 return data
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	778
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	779 def readline(self, size=None):
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	780
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	781 if size is None:
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	782 data = self.reader.readline()
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	783 else:
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	784 data = self.reader.readline(size)
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	785 data, bytesencoded = self.encode(data, self.errors)
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	786 return data
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	787
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	788 def readlines(self, sizehint=None):
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	789
31591 26a80f0af9f3 Ignore sizehint argument. Fixes SF #844561. Marc-André Lemburg <mal@egenix.com> parents: 27618 diff changeset	790 data = self.reader.read()
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	791 data, bytesencoded = self.encode(data, self.errors)
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	792 return data.splitlines(1)
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	793
41501 0c5019afc55b PEP 3114: rename .next() to .__next__() and add next() builtin. Georg Brandl <georg@python.org> parents: 41473 diff changeset	794 def __next__(self):
26082 f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	795
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	796 """ Return the next decoded line from the input stream."""
41501 0c5019afc55b PEP 3114: rename .next() to .__next__() and add next() builtin. Georg Brandl <georg@python.org> parents: 41473 diff changeset	797 data = next(self.reader)
35508 1f2af259968b SF bug #1235646: codecs.StreamRecoder.next() now reencodes the data it reads Walter Dörwald <walter@livinglogic.de> parents: 35415 diff changeset	798 data, bytesencoded = self.encode(data, self.errors)
1f2af259968b SF bug #1235646: codecs.StreamRecoder.next() now reencodes the data it reads Walter Dörwald <walter@livinglogic.de> parents: 35415 diff changeset	799 return data
26082 f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	800
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	801 def __iter__(self):
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	802 return self
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	803
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	804 def write(self, data):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	805
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	806 data, bytesdecoded = self.decode(data, self.errors)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	807 return self.writer.write(data)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	808
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	809 def writelines(self, list):
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	810
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	811 data = ''.join(list)
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	812 data, bytesdecoded = self.decode(data, self.errors)
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	813 return self.writer.write(data)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	814
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	815 def reset(self):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	816
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	817 self.reader.reset()
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	818 self.writer.reset()
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	819
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	820 def __getattr__(self, name,
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	821 getattr=getattr):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	822
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	823 """ Inherit all other methods from the underlying stream.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	824 """
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	825 return getattr(self.stream, name)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	826
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	827 def __enter__(self):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	828 return self
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	829
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	830 def __exit__(self, type, value, tb):
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	831 self.stream.close()
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	832
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	833 ### Shortcuts
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	834
12986 5279a6e00d36 Marc-Andre Lemburg <mal@lemburg.com>: Marc-André Lemburg <mal@egenix.com> parents: 12534 diff changeset	835 def open(filename, mode='rb', encoding=None, errors='strict', buffering=1):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	836
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	837 """ Open an encoded file using the given mode and return
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	838 a wrapped version providing transparent encoding/decoding.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	839
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	840 Note: The wrapped version will only accept the object format
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	841 defined by the codecs, i.e. Unicode objects for most builtin
34761 07745166f571 typo Skip Montanaro <skip@pobox.com> parents: 34753 diff changeset	842 codecs. Output is also codec dependent and will usually be
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	843 Unicode as well.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	844
12986 5279a6e00d36 Marc-Andre Lemburg <mal@lemburg.com>: Marc-André Lemburg <mal@egenix.com> parents: 12534 diff changeset	845 Files are always opened in binary mode, even if no binary mode
27618 74d9604e9353 Fix typos. Walter Dörwald <walter@livinglogic.de> parents: 26906 diff changeset	846 was specified. This is done to avoid data loss due to encodings
12986 5279a6e00d36 Marc-Andre Lemburg <mal@lemburg.com>: Marc-André Lemburg <mal@egenix.com> parents: 12534 diff changeset	847 using 8-bit values. The default file mode is 'rb' meaning to
5279a6e00d36 Marc-Andre Lemburg <mal@lemburg.com>: Marc-André Lemburg <mal@egenix.com> parents: 12534 diff changeset	848 open the file in binary read mode.
5279a6e00d36 Marc-Andre Lemburg <mal@lemburg.com>: Marc-André Lemburg <mal@egenix.com> parents: 12534 diff changeset	849
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	850 encoding specifies the encoding which is to be used for the
27618 74d9604e9353 Fix typos. Walter Dörwald <walter@livinglogic.de> parents: 26906 diff changeset	851 file.
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	852
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	853 errors may be given to define the error handling. It defaults
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	854 to 'strict' which causes ValueErrors to be raised in case an
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	855 encoding error occurs.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	856
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	857 buffering has the same meaning as for the builtin open() API.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	858 It defaults to line buffered.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	859
12389 e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	860 The returned wrapped file object provides an extra attribute
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	861 .encoding which allows querying the used encoding. This
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	862 attribute is only available if an encoding was specified as
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	863 parameter.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	864
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	865 """
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	866 if encoding is not None and \
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	867 'b' not in mode:
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	868 # Force opening of the file in binary mode
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	869 mode = mode + 'b'
44098 0889daf6d0d9 #1535: rename __builtin__ module to builtins. Georg Brandl <georg@python.org> parents: 41606 diff changeset	870 file = builtins.open(filename, mode, buffering)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	871 if encoding is None:
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	872 return file
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	873 info = lookup(encoding)
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	874 srw = StreamReaderWriter(file, info.streamreader, info.streamwriter, errors)
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	875 # Add attributes to simplify introspection
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	876 srw.encoding = encoding
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	877 return srw
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	878
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	879 def EncodedFile(file, data_encoding, file_encoding=None, errors='strict'):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	880
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	881 """ Return a wrapped version of file which provides transparent
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	882 encoding translation.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	883
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	884 Strings written to the wrapped file are interpreted according
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	885 to the given data_encoding and then written to the original
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	886 file as string using file_encoding. The intermediate encoding
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	887 will usually be Unicode but depends on the specified codecs.
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	888
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	889 Strings are read from the file using file_encoding and then
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	890 passed back to the caller as string using data_encoding.
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	891
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	892 If file_encoding is not given, it defaults to data_encoding.
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	893
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	894 errors may be given to define the error handling. It defaults
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	895 to 'strict' which causes ValueErrors to be raised in case an
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	896 encoding error occurs.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	897
12389 e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	898 The returned wrapped file object provides two extra attributes
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	899 .data_encoding and .file_encoding which reflect the given
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	900 parameters of the same name. The attributes can be used for
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	901 introspection by Python programs.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	902
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	903 """
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	904 if file_encoding is None:
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	905 file_encoding = data_encoding
40450 cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	906 data_info = lookup(data_encoding)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	907 file_info = lookup(file_encoding)
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	908 sr = StreamRecoder(file, data_info.encode, data_info.decode,
cc992d75d5b3 Four months of trunk changes (including a few releases...) Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	909 file_info.streamreader, file_info.streamwriter, errors)
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	910 # Add attributes to simplify introspection
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	911 sr.data_encoding = data_encoding
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	912 sr.file_encoding = file_encoding
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	913 return sr
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	914
19981 0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	915 ### Helpers for codec lookup
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	916
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	917 def getencoder(encoding):
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	918
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	919 """ Lookup up the codec for the given encoding and return
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	920 its encoder function.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	921
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	922 Raises a LookupError in case the encoding cannot be found.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	923
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	924 """
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	925 return lookup(encoding).encode
19981 0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	926
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	927 def getdecoder(encoding):
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	928
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	929 """ Lookup up the codec for the given encoding and return
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	930 its decoder function.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	931
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	932 Raises a LookupError in case the encoding cannot be found.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	933
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	934 """
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	935 return lookup(encoding).decode
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	936
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	937 def getincrementalencoder(encoding):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	938
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	939 """ Lookup up the codec for the given encoding and return
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	940 its IncrementalEncoder class or factory function.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	941
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	942 Raises a LookupError in case the encoding cannot be found
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	943 or the codecs doesn't provide an incremental encoder.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	944
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	945 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	946 encoder = lookup(encoding).incrementalencoder
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	947 if encoder is None:
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	948 raise LookupError(encoding)
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	949 return encoder
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	950
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	951 def getincrementaldecoder(encoding):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	952
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	953 """ Lookup up the codec for the given encoding and return
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	954 its IncrementalDecoder class or factory function.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	955
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	956 Raises a LookupError in case the encoding cannot be found
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	957 or the codecs doesn't provide an incremental decoder.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	958
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	959 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	960 decoder = lookup(encoding).incrementaldecoder
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	961 if decoder is None:
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	962 raise LookupError(encoding)
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	963 return decoder
19981 0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	964
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	965 def getreader(encoding):
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	966
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	967 """ Lookup up the codec for the given encoding and return
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	968 its StreamReader class or factory function.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	969
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	970 Raises a LookupError in case the encoding cannot be found.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	971
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	972 """
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	973 return lookup(encoding).streamreader
19981 0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	974
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	975 def getwriter(encoding):
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	976
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	977 """ Lookup up the codec for the given encoding and return
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	978 its StreamWriter class or factory function.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	979
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	980 Raises a LookupError in case the encoding cannot be found.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	981
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	982 """
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	983 return lookup(encoding).streamwriter
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	984
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	985 def iterencode(iterator, encoding, errors='strict', **kwargs):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	986 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	987 Encoding iterator.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	988
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	989 Encodes the input strings from the iterator using a IncrementalEncoder.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	990
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	991 errors and kwargs are passed through to the IncrementalEncoder
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	992 constructor.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	993 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	994 encoder = getincrementalencoder(encoding)(errors, **kwargs)
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	995 for input in iterator:
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	996 output = encoder.encode(input)
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	997 if output:
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	998 yield output
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	999 output = encoder.encode("", True)
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1000 if output:
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1001 yield output
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1002
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1003 def iterdecode(iterator, encoding, errors='strict', **kwargs):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1004 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1005 Decoding iterator.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1006
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1007 Decodes the input strings from the iterator using a IncrementalDecoder.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1008
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1009 errors and kwargs are passed through to the IncrementalDecoder
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1010 constructor.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1011 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1012 decoder = getincrementaldecoder(encoding)(errors, **kwargs)
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1013 for input in iterator:
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1014 output = decoder.decode(input)
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1015 if output:
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1016 yield output
41606 a3e3001b35bf Make the BOM constants in codecs.py bytes. Walter Dörwald <walter@livinglogic.de> parents: 41575 diff changeset	1017 output = decoder.decode(b"", True)
37875 a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1018 if output:
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030 Thomas Wouters <thomas@python.org> parents: 36855 diff changeset	1019 yield output
19981 0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	1020
15706 bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	1021 ### Helpers for charmap-based codecs
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	1022
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	1023 def make_identity_dict(rng):
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	1024
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	1025 """ make_identity_dict(rng) -> dict
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	1026
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	1027 Return a dictionary where elements of the rng sequence are
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	1028 mapped to themselves.
15867 f5a3d98a35bc Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 15706 diff changeset	1029
15706 bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	1030 """
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	1031 res = {}
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	1032 for i in rng:
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	1033 res[i]=i
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	1034 return res
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	1035
17843 3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1036 def make_encoding_map(decoding_map):
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1037
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1038 """ Creates an encoding map from a decoding map.
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1039
27618 74d9604e9353 Fix typos. Walter Dörwald <walter@livinglogic.de> parents: 26906 diff changeset	1040 If a target mapping in the decoding map occurs multiple
17843 3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1041 times, then that target is mapped to None (undefined mapping),
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1042 causing an exception when encountered by the charmap codec
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1043 during translation.
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1044
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1045 One example where this happens is cp875.py which decodes
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1046 multiple character to \u001a.
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1047
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1048 """
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1049 m = {}
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1050 for k,v in decoding_map.items():
23620 a1de0bab5475 SF 563203. Replaced 'has_key()' with 'in'. Raymond Hettinger <python@rcn.com> parents: 22149 diff changeset	1051 if not v in m:
17843 3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1052 m[v] = k
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1053 else:
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1054 m[v] = None
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	1055 return m
17978 5e87a08abb05 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 17843 diff changeset	1056
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	1057 ### error handlers
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	1058
34707 0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	1059 try:
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	1060 strict_errors = lookup_error("strict")
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	1061 ignore_errors = lookup_error("ignore")
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	1062 replace_errors = lookup_error("replace")
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	1063 xmlcharrefreplace_errors = lookup_error("xmlcharrefreplace")
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	1064 backslashreplace_errors = lookup_error("backslashreplace")
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	1065 except LookupError:
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	1066 # In --disable-unicode builds, these error handler are missing
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	1067 strict_errors = None
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	1068 ignore_errors = None
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	1069 replace_errors = None
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	1070 xmlcharrefreplace_errors = None
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	1071 backslashreplace_errors = None
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	1072
18904 5af1ee422575 Add dead imports of modules that are "magically" imported. Martin v. Löwis <martin@v.loewis.de> parents: 17978 diff changeset	1073 # Tell modulefinder that using codecs probably needs the encodings
5af1ee422575 Add dead imports of modules that are "magically" imported. Martin v. Löwis <martin@v.loewis.de> parents: 17978 diff changeset	1074 # package
5af1ee422575 Add dead imports of modules that are "magically" imported. Martin v. Löwis <martin@v.loewis.de> parents: 17978 diff changeset	1075 _false = 0
5af1ee422575 Add dead imports of modules that are "magically" imported. Martin v. Löwis <martin@v.loewis.de> parents: 17978 diff changeset	1076 if _false:
5af1ee422575 Add dead imports of modules that are "magically" imported. Martin v. Löwis <martin@v.loewis.de> parents: 17978 diff changeset	1077 import encodings
5af1ee422575 Add dead imports of modules that are "magically" imported. Martin v. Löwis <martin@v.loewis.de> parents: 17978 diff changeset	1078
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	1079 ### Tests
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	1080
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	1081 if __name__ == '__main__':
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	1082
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	1083 # Make stdout translate Latin-1 output into UTF-8 output
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	1084 sys.stdout = EncodedFile(sys.stdout, 'latin-1', 'utf-8')
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	1085
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	1086 # Have stdin translate Latin-1 input into UTF-8 input
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	1087 sys.stdin = EncodedFile(sys.stdin, 'utf-8', 'latin-1')

Mercurial > cpython-lru_cache-weakref

annotate Lib/codecs.py @ 48715:48dddfd2fbf3