cpython-lru_cache-weakref: Lib/codecs.py annotate

annotate Lib/codecs.py @ 38664:7619216bbc19 2.4

Bug #1502728: Correctly link against librt library on HP-UX. (backport from rev. 46743)

author	Georg Brandl <georg@python.org>
date	Thu, 08 Jun 2006 12:54:16 +0000
parents	72a720dd64fb
children

rev	line source
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	1 """ codecs -- Python Codec Registry, API and helpers.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	2
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	3
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	4 Written by Marc-Andre Lemburg (mal@lemburg.com).
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	5
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	6 (c) Copyright CNRI, All Rights Reserved. NO WARRANTY.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	7
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	8 """#"
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	9
26616 20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	10 import __builtin__, sys
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	11
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	12 ### Registry and builtin stateless codec functions
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	13
12195 bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12051 diff changeset	14 try:
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12051 diff changeset	15 from _codecs import *
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	16 except ImportError, why:
12195 bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12051 diff changeset	17 raise SystemError,\
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12051 diff changeset	18 'Failed to load the builtin codecs: %s' % why
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	19
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	20 __all__ = ["register", "lookup", "open", "EncodedFile", "BOM", "BOM_BE",
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	21 "BOM_LE", "BOM32_BE", "BOM32_LE", "BOM64_BE", "BOM64_LE",
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	22 "BOM_UTF8", "BOM_UTF16", "BOM_UTF16_LE", "BOM_UTF16_BE",
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	23 "BOM_UTF32", "BOM_UTF32_LE", "BOM_UTF32_BE",
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	24 "strict_errors", "ignore_errors", "replace_errors",
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	25 "xmlcharrefreplace_errors",
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	26 "register_error", "lookup_error"]
16070 b5345e99596b added __all__ lists to a number of Python modules Skip Montanaro <skip@pobox.com> parents: 15867 diff changeset	27
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	28 ### Constants
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	29
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	30 #
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	31 # Byte Order Mark (BOM = ZERO WIDTH NO-BREAK SPACE = U+FEFF)
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	32 # and its possible byte string values
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	33 # for UTF8/UTF16/UTF32 output and little/big endian machines
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	34 #
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	35
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	36 # UTF-8
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	37 BOM_UTF8 = '\xef\xbb\xbf'
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	38
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	39 # UTF-16, little endian
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	40 BOM_LE = BOM_UTF16_LE = '\xff\xfe'
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	41
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	42 # UTF-16, big endian
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	43 BOM_BE = BOM_UTF16_BE = '\xfe\xff'
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	44
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	45 # UTF-32, little endian
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	46 BOM_UTF32_LE = '\xff\xfe\x00\x00'
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	47
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	48 # UTF-32, big endian
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	49 BOM_UTF32_BE = '\x00\x00\xfe\xff'
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	50
26616 20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	51 if sys.byteorder == 'little':
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	52
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	53 # UTF-16, native endianness
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	54 BOM = BOM_UTF16 = BOM_UTF16_LE
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	55
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	56 # UTF-32, native endianness
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	57 BOM_UTF32 = BOM_UTF32_LE
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	58
26616 20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	59 else:
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	60
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	61 # UTF-16, native endianness
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	62 BOM = BOM_UTF16 = BOM_UTF16_BE
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	63
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	64 # UTF-32, native endianness
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since Marc-André Lemburg <mal@egenix.com> parents: 26293 diff changeset	65 BOM_UTF32 = BOM_UTF32_BE
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	66
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	67 # Old broken names (don't use in new code)
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	68 BOM32_LE = BOM_UTF16_LE
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	69 BOM32_BE = BOM_UTF16_BE
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	70 BOM64_LE = BOM_UTF32_LE
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23620 diff changeset	71 BOM64_BE = BOM_UTF32_BE
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	72
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	73
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	74 ### Codec base classes (defining the API)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	75
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	76 class Codec:
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	77
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	78 """ Defines the interface for stateless encoders/decoders.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	79
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	80 The .encode()/.decode() methods may use different error
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	81 handling schemes by providing the errors argument. These
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	82 string values are predefined:
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	83
12051 d89556c04ef9 Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12011 diff changeset	84 'strict' - raise a ValueError error (or a subclass)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	85 'ignore' - ignore the character and continue with the next
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	86 'replace' - replace with a suitable replacement character;
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	87 Python will use the official U+FFFD REPLACEMENT
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	88 CHARACTER for the builtin Unicode codecs on
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	89 decoding and '?' on encoding.
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	90 'xmlcharrefreplace' - Replace with the appropriate XML
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	91 character reference (only for encoding).
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	92 'backslashreplace' - Replace with backslashed escape sequences
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	93 (only for encoding).
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	94
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	95 The set of allowed values can be extended via register_error.
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	96
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	97 """
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	98 def encode(self, input, errors='strict'):
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	99
12010 bd1014215f62 "intput" --> "input" (in docstring); noted by Detlef Lannert Fred Drake <fdrake@acm.org> parents: 11990 diff changeset	100 """ Encodes the object input and returns a tuple (output
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	101 object, length consumed).
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	102
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	103 errors defines the error handling to apply. It defaults to
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	104 'strict' handling.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	105
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	106 The method may not store state in the Codec instance. Use
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	107 StreamCodec for codecs which have to keep state in order to
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	108 make encoding/decoding efficient.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	109
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	110 The encoder must be able to handle zero length input and
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	111 return an empty object of the output object type in this
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	112 situation.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	113
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	114 """
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	115 raise NotImplementedError
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	116
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	117 def decode(self, input, errors='strict'):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	118
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	119 """ Decodes the object input and returns a tuple (output
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	120 object, length consumed).
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	121
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	122 input must be an object which provides the bf_getreadbuf
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	123 buffer slot. Python strings, buffer objects and memory
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	124 mapped files are examples of objects providing this slot.
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	125
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	126 errors defines the error handling to apply. It defaults to
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	127 'strict' handling.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	128
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	129 The method may not store state in the Codec instance. Use
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	130 StreamCodec for codecs which have to keep state in order to
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	131 make encoding/decoding efficient.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	132
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	133 The decoder must be able to handle zero length input and
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	134 return an empty object of the output object type in this
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	135 situation.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	136
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	137 """
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	138 raise NotImplementedError
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	139
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	140 #
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	141 # The StreamWriter and StreamReader class provide generic working
19967 0b51e607f333 Fix typo in comment Andrew M. Kuchling <amk@amk.ca> parents: 18936 diff changeset	142 # interfaces which can be used to implement new encoding submodules
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	143 # very easily. See encodings/utf_8.py for an example on how this is
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	144 # done.
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	145 #
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	146
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	147 class StreamWriter(Codec):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	148
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	149 def __init__(self, stream, errors='strict'):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	150
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	151 """ Creates a StreamWriter instance.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	152
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	153 stream must be a file-like object open for writing
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	154 (binary) data.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	155
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	156 The StreamWriter may use different error handling
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	157 schemes by providing the errors keyword argument. These
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	158 parameters are predefined:
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	159
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	160 'strict' - raise a ValueError (or a subclass)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	161 'ignore' - ignore the character and continue with the next
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	162 'replace'- replace with a suitable replacement character
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	163 'xmlcharrefreplace' - Replace with the appropriate XML
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	164 character reference.
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	165 'backslashreplace' - Replace with backslashed escape
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	166 sequences (only for encoding).
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	167
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	168 The set of allowed parameter values can be extended via
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	169 register_error.
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	170 """
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	171 self.stream = stream
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	172 self.errors = errors
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	173
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	174 def write(self, object):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	175
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	176 """ Writes the object's contents encoded to self.stream.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	177 """
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	178 data, consumed = self.encode(object, self.errors)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	179 self.stream.write(data)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	180
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	181 def writelines(self, list):
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	182
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	183 """ Writes the concatenated list of strings to the stream
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	184 using .write().
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	185 """
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	186 self.write(''.join(list))
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	187
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	188 def reset(self):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	189
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	190 """ Flushes and resets the codec buffers used for keeping state.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	191
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	192 Calling this method should ensure that the data on the
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	193 output is put into a clean state, that allows appending
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	194 of new fresh data without having to rescan the whole
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	195 stream to recover state.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	196
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	197 """
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	198 pass
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	199
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	200 def __getattr__(self, name,
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	201 getattr=getattr):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	202
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	203 """ Inherit all other methods from the underlying stream.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	204 """
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	205 return getattr(self.stream, name)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	206
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	207 ###
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	208
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	209 class StreamReader(Codec):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	210
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	211 def __init__(self, stream, errors='strict'):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	212
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	213 """ Creates a StreamReader instance.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	214
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	215 stream must be a file-like object open for reading
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	216 (binary) data.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	217
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	218 The StreamReader may use different error handling
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	219 schemes by providing the errors keyword argument. These
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	220 parameters are predefined:
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	221
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	222 'strict' - raise a ValueError (or a subclass)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	223 'ignore' - ignore the character and continue with the next
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	224 'replace'- replace with a suitable replacement character;
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	225
26293 ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	226 The set of allowed parameter values can be extended via
ff5b510eba6b Add missing documentation for the PEP 293 functionality to Walter Dörwald <walter@livinglogic.de> parents: 26082 diff changeset	227 register_error.
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	228 """
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	229 self.stream = stream
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	230 self.errors = errors
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	231 self.bytebuffer = ""
35270 e789703fe4be Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34926 diff changeset	232 # For str->str decoding this will stay a str
e789703fe4be Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34926 diff changeset	233 # For str->unicode decoding the first read will promote it to unicode
e789703fe4be Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34926 diff changeset	234 self.charbuffer = ""
35590 4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	235 self.linebuffer = None
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	236
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	237 def decode(self, input, errors='strict'):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	238 raise NotImplementedError
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	239
35416 4896f8334bff Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35270 diff changeset	240 def read(self, size=-1, chars=-1, firstline=False):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	241
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	242 """ Decodes data from the stream self.stream and returns the
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	243 resulting object.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	244
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	245 chars indicates the number of characters to read from the
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	246 stream. read() will never return more than chars
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	247 characters, but it might return less, if there are not enough
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	248 characters available.
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	249
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	250 size indicates the approximate maximum number of bytes to
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	251 read from the stream for decoding purposes. The decoder
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	252 can modify this setting as appropriate. The default value
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	253 -1 indicates to read and decode as much as possible. size
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	254 is intended to prevent having to decode huge files in one
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	255 step.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	256
35416 4896f8334bff Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35270 diff changeset	257 If firstline is true, and a UnicodeDecodeError happens
4896f8334bff Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35270 diff changeset	258 after the first line terminator in the input only the first line
4896f8334bff Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35270 diff changeset	259 will be returned, the rest of the input will be kept until the
4896f8334bff Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35270 diff changeset	260 next call to read().
4896f8334bff Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35270 diff changeset	261
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	262 The method should use a greedy read strategy meaning that
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	263 it should read as much data as is allowed within the
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	264 definition of the encoding and the given size, e.g. if
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	265 optional encoding endings or state markers are available
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	266 on the stream, these should be read too.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	267 """
35590 4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	268 # If we have lines cached, first merge them back into characters
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	269 if self.linebuffer:
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	270 self.charbuffer = "".join(self.linebuffer)
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	271 self.linebuffer = None
37171 72a720dd64fb Mechanical whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 36856 diff changeset	272
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	273 # read until we get the required number of characters (if available)
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	274 while True:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	275 # can the request can be satisfied from the character buffer?
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	276 if chars < 0:
36856 fe0736ccba75 Backport revision 42872: Walter Dörwald <walter@livinglogic.de> parents: 35590 diff changeset	277 if size < 0:
fe0736ccba75 Backport revision 42872: Walter Dörwald <walter@livinglogic.de> parents: 35590 diff changeset	278 if self.charbuffer:
fe0736ccba75 Backport revision 42872: Walter Dörwald <walter@livinglogic.de> parents: 35590 diff changeset	279 break
fe0736ccba75 Backport revision 42872: Walter Dörwald <walter@livinglogic.de> parents: 35590 diff changeset	280 elif len(self.charbuffer) >= size:
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	281 break
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	282 else:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	283 if len(self.charbuffer) >= chars:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	284 break
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	285 # we need more data
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	286 if size < 0:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	287 newdata = self.stream.read()
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	288 else:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	289 newdata = self.stream.read(size)
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	290 # decode bytes (those remaining from the last call included)
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	291 data = self.bytebuffer + newdata
35416 4896f8334bff Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35270 diff changeset	292 try:
4896f8334bff Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35270 diff changeset	293 newchars, decodedbytes = self.decode(data, self.errors)
4896f8334bff Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35270 diff changeset	294 except UnicodeDecodeError, exc:
4896f8334bff Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35270 diff changeset	295 if firstline:
4896f8334bff Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35270 diff changeset	296 newchars, decodedbytes = self.decode(data[:exc.start], self.errors)
4896f8334bff Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35270 diff changeset	297 lines = newchars.splitlines(True)
4896f8334bff Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35270 diff changeset	298 if len(lines)<=1:
4896f8334bff Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35270 diff changeset	299 raise
4896f8334bff Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35270 diff changeset	300 else:
4896f8334bff Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35270 diff changeset	301 raise
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	302 # keep undecoded bytes until the next call
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	303 self.bytebuffer = data[decodedbytes:]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	304 # put new characters in the character buffer
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	305 self.charbuffer += newchars
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	306 # there was no data available
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	307 if not newdata:
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	308 break
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	309 if chars < 0:
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	310 # Return everything we've got
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	311 result = self.charbuffer
35270 e789703fe4be Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34926 diff changeset	312 self.charbuffer = ""
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	313 else:
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	314 # Return the first chars characters
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	315 result = self.charbuffer[:chars]
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	316 self.charbuffer = self.charbuffer[chars:]
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	317 return result
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	318
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	319 def readline(self, size=None, keepends=True):
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	320
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	321 """ Read one line from the input stream and return the
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	322 decoded data.
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	323
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	324 size, if given, is passed as size argument to the
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	325 read() method.
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	326
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	327 """
35590 4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	328 # If we have lines cached from an earlier read, return
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	329 # them unconditionally
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	330 if self.linebuffer:
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	331 line = self.linebuffer[0]
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	332 del self.linebuffer[0]
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	333 if len(self.linebuffer) == 1:
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	334 # revert to charbuffer mode; we might need more data
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	335 # next time
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	336 self.charbuffer = self.linebuffer[0]
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	337 self.linebuffer = None
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	338 if not keepends:
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	339 line = line.splitlines(False)[0]
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	340 return line
37171 72a720dd64fb Mechanical whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 36856 diff changeset	341
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	342 readsize = size or 72
35270 e789703fe4be Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34926 diff changeset	343 line = ""
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	344 # If size is given, we call read() only once
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	345 while True:
35416 4896f8334bff Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 35270 diff changeset	346 data = self.read(readsize, firstline=True)
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	347 if data:
34926 7f72109dfb4f Backport checkin (and the appropriate fix to the test): Walter Dörwald <walter@livinglogic.de> parents: 34867 diff changeset	348 # If we're at a "\r" read one extra character (which might
7f72109dfb4f Backport checkin (and the appropriate fix to the test): Walter Dörwald <walter@livinglogic.de> parents: 34867 diff changeset	349 # be a "\n") to get a proper line ending. If the stream is
7f72109dfb4f Backport checkin (and the appropriate fix to the test): Walter Dörwald <walter@livinglogic.de> parents: 34867 diff changeset	350 # temporarily exhausted we return the wrong line ending.
35270 e789703fe4be Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34926 diff changeset	351 if data.endswith("\r"):
34867 da460841b63c Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34753 diff changeset	352 data += self.read(size=1, chars=1)
da460841b63c Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34753 diff changeset	353
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	354 line += data
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	355 lines = line.splitlines(True)
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	356 if lines:
35590 4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	357 if len(lines) > 1:
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	358 # More than one line result; the first line is a full line
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	359 # to return
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	360 line = lines[0]
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	361 del lines[0]
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	362 if len(lines) > 1:
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	363 # cache the remaining lines
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	364 lines[-1] += self.charbuffer
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	365 self.linebuffer = lines
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	366 self.charbuffer = None
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	367 else:
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	368 # only one remaining line, put it back into charbuffer
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	369 self.charbuffer = lines[0] + self.charbuffer
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	370 if not keepends:
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	371 line = line.splitlines(False)[0]
4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	372 break
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	373 line0withend = lines[0]
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	374 line0withoutend = lines[0].splitlines(False)[0]
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	375 if line0withend != line0withoutend: # We really have a line end
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	376 # Put the rest back together and keep it until the next call
35270 e789703fe4be Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34926 diff changeset	377 self.charbuffer = "".join(lines[1:]) + self.charbuffer
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	378 if keepends:
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	379 line = line0withend
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	380 else:
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	381 line = line0withoutend
34387 f25dcd7f3e48 Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	382 break
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	383 # we didn't get anything or this was our only try
34387 f25dcd7f3e48 Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34214 diff changeset	384 if not data or size is not None:
34214 abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	385 if line and not keepends:
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	386 line = line.splitlines(False)[0]
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	387 break
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	388 if readsize<8000:
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	389 readsize *= 2
abe3145dde4e Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 33782 diff changeset	390 return line
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	391
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	392 def readlines(self, sizehint=None, keepends=True):
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	393
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	394 """ Read all lines available on the input stream
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	395 and return them as list of lines.
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	396
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	397 Line breaks are implemented using the codec's decoder
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	398 method and are included in the list entries.
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	399
31591 26a80f0af9f3 Ignore sizehint argument. Fixes SF #844561. Marc-André Lemburg <mal@egenix.com> parents: 27618 diff changeset	400 sizehint, if given, is ignored since there is no efficient
26a80f0af9f3 Ignore sizehint argument. Fixes SF #844561. Marc-André Lemburg <mal@egenix.com> parents: 27618 diff changeset	401 way to finding the true end-of-line.
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	402
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	403 """
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 31591 diff changeset	404 data = self.read()
33782 a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines() Hye-Shik Chang <hyeshik@gmail.com> parents: 33428 diff changeset	405 return data.splitlines(keepends)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	406
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	407 def reset(self):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	408
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	409 """ Resets the codec buffers used for keeping state.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	410
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	411 Note that no stream repositioning should take place.
13620 5b727ab37092 Spelling fixes supplied by Rob W. W. Hooft. All these are fixes in either Thomas Wouters <thomas@python.org> parents: 12986 diff changeset	412 This method is primarily intended to be able to recover
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	413 from decoding errors.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	414
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	415 """
34751 962800ee30ef Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34707 diff changeset	416 self.bytebuffer = ""
962800ee30ef Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34707 diff changeset	417 self.charbuffer = u""
35590 4806cb009a9c Patch #1268314: Cache lines in StreamReader.readlines for performance. Martin v. Löwis <martin@v.loewis.de> parents: 35509 diff changeset	418 self.linebuffer = None
34751 962800ee30ef Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34707 diff changeset	419
34753 11b4350b5879 Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34751 diff changeset	420 def seek(self, offset, whence=0):
34751 962800ee30ef Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34707 diff changeset	421 """ Set the input stream's current position.
962800ee30ef Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34707 diff changeset	422
962800ee30ef Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34707 diff changeset	423 Resets the codec buffers used for keeping state.
962800ee30ef Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34707 diff changeset	424 """
962800ee30ef Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34707 diff changeset	425 self.reset()
962800ee30ef Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 34707 diff changeset	426 self.stream.seek(offset, whence)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	427
26082 f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	428 def next(self):
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	429
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	430 """ Return the next decoded line from the input stream."""
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	431 line = self.readline()
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	432 if line:
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	433 return line
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	434 raise StopIteration
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	435
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	436 def __iter__(self):
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	437 return self
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	438
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	439 def __getattr__(self, name,
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	440 getattr=getattr):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	441
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	442 """ Inherit all other methods from the underlying stream.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	443 """
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	444 return getattr(self.stream, name)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	445
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	446 ###
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	447
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	448 class StreamReaderWriter:
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	449
12389 e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	450 """ StreamReaderWriter instances allow wrapping streams which
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	451 work in both read and write modes.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	452
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	453 The design is such that one can use the factory functions
13620 5b727ab37092 Spelling fixes supplied by Rob W. W. Hooft. All these are fixes in either Thomas Wouters <thomas@python.org> parents: 12986 diff changeset	454 returned by the codec.lookup() function to construct the
12389 e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	455 instance.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	456
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	457 """
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	458 # Optional attributes set by the file wrappers below
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	459 encoding = 'unknown'
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	460
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	461 def __init__(self, stream, Reader, Writer, errors='strict'):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	462
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	463 """ Creates a StreamReaderWriter instance.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	464
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	465 stream must be a Stream-like object.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	466
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	467 Reader, Writer must be factory functions or classes
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	468 providing the StreamReader, StreamWriter interface resp.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	469
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	470 Error handling is done in the same way as defined for the
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	471 StreamWriter/Readers.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	472
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	473 """
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	474 self.stream = stream
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	475 self.reader = Reader(stream, errors)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	476 self.writer = Writer(stream, errors)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	477 self.errors = errors
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	478
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	479 def read(self, size=-1):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	480
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	481 return self.reader.read(size)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	482
12534 189298a1fab4 Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12389 diff changeset	483 def readline(self, size=None):
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	484
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	485 return self.reader.readline(size)
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	486
12534 189298a1fab4 Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12389 diff changeset	487 def readlines(self, sizehint=None):
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	488
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	489 return self.reader.readlines(sizehint)
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	490
26082 f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	491 def next(self):
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	492
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	493 """ Return the next decoded line from the input stream."""
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	494 return self.reader.next()
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	495
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	496 def __iter__(self):
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	497 return self
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	498
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	499 def write(self, data):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	500
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	501 return self.writer.write(data)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	502
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	503 def writelines(self, list):
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	504
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	505 return self.writer.writelines(list)
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	506
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	507 def reset(self):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	508
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	509 self.reader.reset()
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	510 self.writer.reset()
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	511
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	512 def __getattr__(self, name,
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	513 getattr=getattr):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	514
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	515 """ Inherit all other methods from the underlying stream.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	516 """
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	517 return getattr(self.stream, name)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	518
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	519 ###
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	520
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	521 class StreamRecoder:
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	522
12389 e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	523 """ StreamRecoder instances provide a frontend - backend
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	524 view of encoding data.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	525
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	526 They use the complete set of APIs returned by the
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	527 codecs.lookup() function to implement their task.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	528
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	529 Data written to the stream is first decoded into an
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	530 intermediate format (which is dependent on the given codec
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	531 combination) and then written to the stream using an instance
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	532 of the provided Writer class.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	533
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	534 In the other direction, data is read from the stream using a
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	535 Reader instance and then return encoded data to the caller.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	536
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	537 """
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	538 # Optional attributes set by the file wrappers below
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	539 data_encoding = 'unknown'
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	540 file_encoding = 'unknown'
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	541
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	542 def __init__(self, stream, encode, decode, Reader, Writer,
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	543 errors='strict'):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	544
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	545 """ Creates a StreamRecoder instance which implements a two-way
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	546 conversion: encode and decode work on the frontend (the
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	547 input to .read() and output of .write()) while
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	548 Reader and Writer work on the backend (reading and
12011 4fa25a292273 Oops, another in the same file; I should read the mail fully before Fred Drake <fdrake@acm.org> parents: 12010 diff changeset	549 writing to the stream).
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	550
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	551 You can use these objects to do transparent direct
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	552 recodings from e.g. latin-1 to utf-8 and back.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	553
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	554 stream must be a file-like object.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	555
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	556 encode, decode must adhere to the Codec interface, Reader,
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	557 Writer must be factory functions or classes providing the
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	558 StreamReader, StreamWriter interface resp.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	559
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	560 encode and decode are needed for the frontend translation,
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	561 Reader and Writer for the backend translation. Unicode is
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	562 used as intermediate encoding.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	563
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	564 Error handling is done in the same way as defined for the
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	565 StreamWriter/Readers.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	566
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	567 """
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	568 self.stream = stream
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	569 self.encode = encode
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	570 self.decode = decode
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	571 self.reader = Reader(stream, errors)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	572 self.writer = Writer(stream, errors)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	573 self.errors = errors
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	574
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	575 def read(self, size=-1):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	576
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	577 data = self.reader.read(size)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	578 data, bytesencoded = self.encode(data, self.errors)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	579 return data
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	580
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	581 def readline(self, size=None):
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	582
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	583 if size is None:
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	584 data = self.reader.readline()
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	585 else:
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	586 data = self.reader.readline(size)
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	587 data, bytesencoded = self.encode(data, self.errors)
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	588 return data
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	589
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	590 def readlines(self, sizehint=None):
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	591
31591 26a80f0af9f3 Ignore sizehint argument. Fixes SF #844561. Marc-André Lemburg <mal@egenix.com> parents: 27618 diff changeset	592 data = self.reader.read()
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	593 data, bytesencoded = self.encode(data, self.errors)
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	594 return data.splitlines(1)
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	595
26082 f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	596 def next(self):
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	597
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	598 """ Return the next decoded line from the input stream."""
35509 0a28d48636c8 Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 35416 diff changeset	599 data = self.reader.next()
0a28d48636c8 Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 35416 diff changeset	600 data, bytesencoded = self.encode(data, self.errors)
0a28d48636c8 Backport checkin: Walter Dörwald <walter@livinglogic.de> parents: 35416 diff changeset	601 return data
26082 f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	602
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	603 def __iter__(self):
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	604 return self
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter Walter Dörwald <walter@livinglogic.de> parents: 25091 diff changeset	605
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	606 def write(self, data):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	607
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	608 data, bytesdecoded = self.decode(data, self.errors)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	609 return self.writer.write(data)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	610
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	611 def writelines(self, list):
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	612
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	613 data = ''.join(list)
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	614 data, bytesdecoded = self.decode(data, self.errors)
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	615 return self.writer.write(data)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	616
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	617 def reset(self):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	618
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	619 self.reader.reset()
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	620 self.writer.reset()
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	621
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	622 def __getattr__(self, name,
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	623 getattr=getattr):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	624
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	625 """ Inherit all other methods from the underlying stream.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	626 """
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	627 return getattr(self.stream, name)
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	628
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	629 ### Shortcuts
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	630
12986 5279a6e00d36 Marc-Andre Lemburg <mal@lemburg.com>: Marc-André Lemburg <mal@egenix.com> parents: 12534 diff changeset	631 def open(filename, mode='rb', encoding=None, errors='strict', buffering=1):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	632
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	633 """ Open an encoded file using the given mode and return
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	634 a wrapped version providing transparent encoding/decoding.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	635
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	636 Note: The wrapped version will only accept the object format
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	637 defined by the codecs, i.e. Unicode objects for most builtin
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	638 codecs. Output is also codec dependent and will usually by
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	639 Unicode as well.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	640
12986 5279a6e00d36 Marc-Andre Lemburg <mal@lemburg.com>: Marc-André Lemburg <mal@egenix.com> parents: 12534 diff changeset	641 Files are always opened in binary mode, even if no binary mode
27618 74d9604e9353 Fix typos. Walter Dörwald <walter@livinglogic.de> parents: 26906 diff changeset	642 was specified. This is done to avoid data loss due to encodings
12986 5279a6e00d36 Marc-Andre Lemburg <mal@lemburg.com>: Marc-André Lemburg <mal@egenix.com> parents: 12534 diff changeset	643 using 8-bit values. The default file mode is 'rb' meaning to
5279a6e00d36 Marc-Andre Lemburg <mal@lemburg.com>: Marc-André Lemburg <mal@egenix.com> parents: 12534 diff changeset	644 open the file in binary read mode.
5279a6e00d36 Marc-Andre Lemburg <mal@lemburg.com>: Marc-André Lemburg <mal@egenix.com> parents: 12534 diff changeset	645
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	646 encoding specifies the encoding which is to be used for the
27618 74d9604e9353 Fix typos. Walter Dörwald <walter@livinglogic.de> parents: 26906 diff changeset	647 file.
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	648
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	649 errors may be given to define the error handling. It defaults
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	650 to 'strict' which causes ValueErrors to be raised in case an
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	651 encoding error occurs.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	652
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	653 buffering has the same meaning as for the builtin open() API.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	654 It defaults to line buffered.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	655
12389 e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	656 The returned wrapped file object provides an extra attribute
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	657 .encoding which allows querying the used encoding. This
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	658 attribute is only available if an encoding was specified as
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	659 parameter.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	660
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	661 """
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	662 if encoding is not None and \
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	663 'b' not in mode:
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	664 # Force opening of the file in binary mode
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	665 mode = mode + 'b'
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	666 file = __builtin__.open(filename, mode, buffering)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	667 if encoding is None:
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	668 return file
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	669 (e, d, sr, sw) = lookup(encoding)
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	670 srw = StreamReaderWriter(file, sr, sw, errors)
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	671 # Add attributes to simplify introspection
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	672 srw.encoding = encoding
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	673 return srw
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	674
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	675 def EncodedFile(file, data_encoding, file_encoding=None, errors='strict'):
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	676
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	677 """ Return a wrapped version of file which provides transparent
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	678 encoding translation.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	679
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	680 Strings written to the wrapped file are interpreted according
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	681 to the given data_encoding and then written to the original
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	682 file as string using file_encoding. The intermediate encoding
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	683 will usually be Unicode but depends on the specified codecs.
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	684
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	685 Strings are read from the file using file_encoding and then
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	686 passed back to the caller as string using data_encoding.
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	687
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	688 If file_encoding is not given, it defaults to data_encoding.
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	689
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	690 errors may be given to define the error handling. It defaults
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	691 to 'strict' which causes ValueErrors to be raised in case an
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	692 encoding error occurs.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	693
12389 e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	694 The returned wrapped file object provides two extra attributes
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	695 .data_encoding and .file_encoding which reflect the given
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	696 parameters of the same name. The attributes can be used for
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	697 introspection by Python programs.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12373 diff changeset	698
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	699 """
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	700 if file_encoding is None:
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	701 file_encoding = data_encoding
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	702 encode, decode = lookup(data_encoding)[:2]
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	703 Reader, Writer = lookup(file_encoding)[2:]
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	704 sr = StreamRecoder(file,
17839 c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space Tim Peters <tim.peters@gmail.com> parents: 16070 diff changeset	705 encode, decode, Reader, Writer,
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	706 errors)
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	707 # Add attributes to simplify introspection
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	708 sr.data_encoding = data_encoding
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	709 sr.file_encoding = file_encoding
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	710 return sr
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	711
19981 0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	712 ### Helpers for codec lookup
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	713
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	714 def getencoder(encoding):
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	715
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	716 """ Lookup up the codec for the given encoding and return
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	717 its encoder function.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	718
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	719 Raises a LookupError in case the encoding cannot be found.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	720
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	721 """
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	722 return lookup(encoding)[0]
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	723
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	724 def getdecoder(encoding):
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	725
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	726 """ Lookup up the codec for the given encoding and return
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	727 its decoder function.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	728
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	729 Raises a LookupError in case the encoding cannot be found.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	730
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	731 """
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	732 return lookup(encoding)[1]
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	733
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	734 def getreader(encoding):
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	735
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	736 """ Lookup up the codec for the given encoding and return
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	737 its StreamReader class or factory function.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	738
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	739 Raises a LookupError in case the encoding cannot be found.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	740
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	741 """
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	742 return lookup(encoding)[2]
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	743
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	744 def getwriter(encoding):
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	745
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	746 """ Lookup up the codec for the given encoding and return
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	747 its StreamWriter class or factory function.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	748
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	749 Raises a LookupError in case the encoding cannot be found.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	750
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	751 """
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	752 return lookup(encoding)[3]
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow. Marc-André Lemburg <mal@egenix.com> parents: 19967 diff changeset	753
15706 bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	754 ### Helpers for charmap-based codecs
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	755
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	756 def make_identity_dict(rng):
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	757
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	758 """ make_identity_dict(rng) -> dict
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	759
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	760 Return a dictionary where elements of the rng sequence are
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	761 mapped to themselves.
15867 f5a3d98a35bc Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 15706 diff changeset	762
15706 bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	763 """
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	764 res = {}
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	765 for i in rng:
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	766 res[i]=i
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	767 return res
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap Marc-André Lemburg <mal@egenix.com> parents: 15498 diff changeset	768
17843 3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	769 def make_encoding_map(decoding_map):
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	770
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	771 """ Creates an encoding map from a decoding map.
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	772
27618 74d9604e9353 Fix typos. Walter Dörwald <walter@livinglogic.de> parents: 26906 diff changeset	773 If a target mapping in the decoding map occurs multiple
17843 3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	774 times, then that target is mapped to None (undefined mapping),
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	775 causing an exception when encountered by the charmap codec
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	776 during translation.
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	777
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	778 One example where this happens is cp875.py which decodes
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	779 multiple character to \u001a.
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	780
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	781 """
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	782 m = {}
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	783 for k,v in decoding_map.items():
23620 a1de0bab5475 SF 563203. Replaced 'has_key()' with 'in'. Raymond Hettinger <python@rcn.com> parents: 22149 diff changeset	784 if not v in m:
17843 3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	785 m[v] = k
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	786 else:
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	787 m[v] = None
3e256f9d99ad Moved the encoding map building logic from the individual mapping Marc-André Lemburg <mal@egenix.com> parents: 17839 diff changeset	788 return m
17978 5e87a08abb05 Whitespace normalization. Tim Peters <tim.peters@gmail.com> parents: 17843 diff changeset	789
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	790 ### error handlers
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	791
34707 0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	792 try:
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	793 strict_errors = lookup_error("strict")
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	794 ignore_errors = lookup_error("ignore")
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	795 replace_errors = lookup_error("replace")
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	796 xmlcharrefreplace_errors = lookup_error("xmlcharrefreplace")
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	797 backslashreplace_errors = lookup_error("backslashreplace")
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	798 except LookupError:
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	799 # In --disable-unicode builds, these error handler are missing
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	800 strict_errors = None
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	801 ignore_errors = None
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	802 replace_errors = None
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	803 xmlcharrefreplace_errors = None
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607. Martin v. Löwis <martin@v.loewis.de> parents: 34387 diff changeset	804 backslashreplace_errors = None
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	805
18904 5af1ee422575 Add dead imports of modules that are "magically" imported. Martin v. Löwis <martin@v.loewis.de> parents: 17978 diff changeset	806 # Tell modulefinder that using codecs probably needs the encodings
5af1ee422575 Add dead imports of modules that are "magically" imported. Martin v. Löwis <martin@v.loewis.de> parents: 17978 diff changeset	807 # package
5af1ee422575 Add dead imports of modules that are "magically" imported. Martin v. Löwis <martin@v.loewis.de> parents: 17978 diff changeset	808 _false = 0
5af1ee422575 Add dead imports of modules that are "magically" imported. Martin v. Löwis <martin@v.loewis.de> parents: 17978 diff changeset	809 if _false:
5af1ee422575 Add dead imports of modules that are "magically" imported. Martin v. Löwis <martin@v.loewis.de> parents: 17978 diff changeset	810 import encodings
5af1ee422575 Add dead imports of modules that are "magically" imported. Martin v. Löwis <martin@v.loewis.de> parents: 17978 diff changeset	811
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	812 ### Tests
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	813
11990 f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	814 if __name__ == '__main__':
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by Guido van Rossum <guido@python.org> parents: diff changeset	815
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	816 # Make stdout translate Latin-1 output into UTF-8 output
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	817 sys.stdout = EncodedFile(sys.stdout, 'latin-1', 'utf-8')
12373 d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add Guido van Rossum <guido@python.org> parents: 12367 diff changeset	818
12367 677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	819 # Have stdin translate Latin-1 input into UTF-8 input
677e2147829a Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12195 diff changeset	820 sys.stdin = EncodedFile(sys.stdin, 'utf-8', 'latin-1')

Mercurial > cpython-lru_cache-weakref

annotate Lib/codecs.py @ 38664:7619216bbc19 2.4