Mercurial > cpython-lru_cache-weakref

annotate Lib/codecs.py @ 64166:d0e9febfa00a 3.1

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Merged revisions 84172 via svnmerge from svn+ssh://pythondev@svn.python.org/python/branches/py3k ........ r84172 | martin.v.loewis | 2010-08-18 18:12:23 +0200 (Mi, 18 Aug 2010) | 2 lines Restore GIL in nis_cat in case of error. ........
author Martin v. Löwis <martin@v.loewis.de>
date Thu, 19 Aug 2010 09:03:03 +0000
parents cd4936691163
children 7e4833764c88
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1 """ codecs -- Python Codec Registry, API and helpers.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
2
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
3
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
4 Written by Marc-Andre Lemburg (mal@lemburg.com).
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
5
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
6 (c) Copyright CNRI, All Rights Reserved. NO WARRANTY.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
7
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
8 """#"
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
9
44098
0889daf6d0d9 #1535: rename __builtin__ module to builtins.
Georg Brandl <georg@python.org>
parents: 41606
diff changeset
10 import builtins, sys
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
11
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
12 ### Registry and builtin stateless codec functions
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
13
12195
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12051
diff changeset
14 try:
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12051
diff changeset
15 from _codecs import *
40587
6b18a095c2e7 SF patch 1631942 by Collin Winter:
Guido van Rossum <guido@python.org>
parents: 40450
diff changeset
16 except ImportError as why:
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
17 raise SystemError('Failed to load the builtin codecs: %s' % why)
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
18
17839
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
19 __all__ = ["register", "lookup", "open", "EncodedFile", "BOM", "BOM_BE",
23658
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE,
Walter Dörwald <walter@livinglogic.de>
parents: 23620
diff changeset
20 "BOM_LE", "BOM32_BE", "BOM32_LE", "BOM64_BE", "BOM64_LE",
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE,
Walter Dörwald <walter@livinglogic.de>
parents: 23620
diff changeset
21 "BOM_UTF8", "BOM_UTF16", "BOM_UTF16_LE", "BOM_UTF16_BE",
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 23658
diff changeset
22 "BOM_UTF32", "BOM_UTF32_LE", "BOM_UTF32_BE",
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 23658
diff changeset
23 "strict_errors", "ignore_errors", "replace_errors",
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 23658
diff changeset
24 "xmlcharrefreplace_errors",
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 23658
diff changeset
25 "register_error", "lookup_error"]
16070
b5345e99596b added __all__ lists to a number of Python modules
Skip Montanaro <skip@pobox.com>
parents: 15867
diff changeset
26
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
27 ### Constants
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
28
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
29 #
23658
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE,
Walter Dörwald <walter@livinglogic.de>
parents: 23620
diff changeset
30 # Byte Order Mark (BOM = ZERO WIDTH NO-BREAK SPACE = U+FEFF)
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE,
Walter Dörwald <walter@livinglogic.de>
parents: 23620
diff changeset
31 # and its possible byte string values
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE,
Walter Dörwald <walter@livinglogic.de>
parents: 23620
diff changeset
32 # for UTF8/UTF16/UTF32 output and little/big endian machines
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
33 #
23658
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE,
Walter Dörwald <walter@livinglogic.de>
parents: 23620
diff changeset
34
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE,
Walter Dörwald <walter@livinglogic.de>
parents: 23620
diff changeset
35 # UTF-8
41606
a3e3001b35bf Make the BOM constants in codecs.py bytes.
Walter Dörwald <walter@livinglogic.de>
parents: 41575
diff changeset
36 BOM_UTF8 = b'\xef\xbb\xbf'
23658
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE,
Walter Dörwald <walter@livinglogic.de>
parents: 23620
diff changeset
37
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE,
Walter Dörwald <walter@livinglogic.de>
parents: 23620
diff changeset
38 # UTF-16, little endian
41606
a3e3001b35bf Make the BOM constants in codecs.py bytes.
Walter Dörwald <walter@livinglogic.de>
parents: 41575
diff changeset
39 BOM_LE = BOM_UTF16_LE = b'\xff\xfe'
23658
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE,
Walter Dörwald <walter@livinglogic.de>
parents: 23620
diff changeset
40
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE,
Walter Dörwald <walter@livinglogic.de>
parents: 23620
diff changeset
41 # UTF-16, big endian
41606
a3e3001b35bf Make the BOM constants in codecs.py bytes.
Walter Dörwald <walter@livinglogic.de>
parents: 41575
diff changeset
42 BOM_BE = BOM_UTF16_BE = b'\xfe\xff'
23658
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE,
Walter Dörwald <walter@livinglogic.de>
parents: 23620
diff changeset
43
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE,
Walter Dörwald <walter@livinglogic.de>
parents: 23620
diff changeset
44 # UTF-32, little endian
41606
a3e3001b35bf Make the BOM constants in codecs.py bytes.
Walter Dörwald <walter@livinglogic.de>
parents: 41575
diff changeset
45 BOM_UTF32_LE = b'\xff\xfe\x00\x00'
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
46
23658
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE,
Walter Dörwald <walter@livinglogic.de>
parents: 23620
diff changeset
47 # UTF-32, big endian
41606
a3e3001b35bf Make the BOM constants in codecs.py bytes.
Walter Dörwald <walter@livinglogic.de>
parents: 41575
diff changeset
48 BOM_UTF32_BE = b'\x00\x00\xfe\xff'
23658
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE,
Walter Dörwald <walter@livinglogic.de>
parents: 23620
diff changeset
49
26616
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since
Marc-André Lemburg <mal@egenix.com>
parents: 26293
diff changeset
50 if sys.byteorder == 'little':
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since
Marc-André Lemburg <mal@egenix.com>
parents: 26293
diff changeset
51
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since
Marc-André Lemburg <mal@egenix.com>
parents: 26293
diff changeset
52 # UTF-16, native endianness
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since
Marc-André Lemburg <mal@egenix.com>
parents: 26293
diff changeset
53 BOM = BOM_UTF16 = BOM_UTF16_LE
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since
Marc-André Lemburg <mal@egenix.com>
parents: 26293
diff changeset
54
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since
Marc-André Lemburg <mal@egenix.com>
parents: 26293
diff changeset
55 # UTF-32, native endianness
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since
Marc-André Lemburg <mal@egenix.com>
parents: 26293
diff changeset
56 BOM_UTF32 = BOM_UTF32_LE
23658
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE,
Walter Dörwald <walter@livinglogic.de>
parents: 23620
diff changeset
57
26616
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since
Marc-André Lemburg <mal@egenix.com>
parents: 26293
diff changeset
58 else:
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since
Marc-André Lemburg <mal@egenix.com>
parents: 26293
diff changeset
59
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since
Marc-André Lemburg <mal@egenix.com>
parents: 26293
diff changeset
60 # UTF-16, native endianness
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since
Marc-André Lemburg <mal@egenix.com>
parents: 26293
diff changeset
61 BOM = BOM_UTF16 = BOM_UTF16_BE
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since
Marc-André Lemburg <mal@egenix.com>
parents: 26293
diff changeset
62
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since
Marc-André Lemburg <mal@egenix.com>
parents: 26293
diff changeset
63 # UTF-32, native endianness
20e0755f3554 Patch to make _codecs a builtin module. This is necessary since
Marc-André Lemburg <mal@egenix.com>
parents: 26293
diff changeset
64 BOM_UTF32 = BOM_UTF32_BE
23658
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE,
Walter Dörwald <walter@livinglogic.de>
parents: 23620
diff changeset
65
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE,
Walter Dörwald <walter@livinglogic.de>
parents: 23620
diff changeset
66 # Old broken names (don't use in new code)
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE,
Walter Dörwald <walter@livinglogic.de>
parents: 23620
diff changeset
67 BOM32_LE = BOM_UTF16_LE
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE,
Walter Dörwald <walter@livinglogic.de>
parents: 23620
diff changeset
68 BOM32_BE = BOM_UTF16_BE
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE,
Walter Dörwald <walter@livinglogic.de>
parents: 23620
diff changeset
69 BOM64_LE = BOM_UTF32_LE
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE,
Walter Dörwald <walter@livinglogic.de>
parents: 23620
diff changeset
70 BOM64_BE = BOM_UTF32_BE
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
71
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
72
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
73 ### Codec base classes (defining the API)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
74
37875
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
75 class CodecInfo(tuple):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
76
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
77 def __new__(cls, encode, decode, streamreader=None, streamwriter=None,
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
78 incrementalencoder=None, incrementaldecoder=None, name=None):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
79 self = tuple.__new__(cls, (encode, decode, streamreader, streamwriter))
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
80 self.name = name
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
81 self.encode = encode
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
82 self.decode = decode
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
83 self.incrementalencoder = incrementalencoder
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
84 self.incrementaldecoder = incrementaldecoder
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
85 self.streamwriter = streamwriter
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
86 self.streamreader = streamreader
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
87 return self
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
88
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
89 def __repr__(self):
41473
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
90 return "<%s.%s object for encoding %s at 0x%x>" % \
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
91 (self.__class__.__module__, self.__class__.__name__,
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
92 self.name, id(self))
37875
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
93
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
94 class Codec:
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
95
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
96 """ Defines the interface for stateless encoders/decoders.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
97
26293
ff5b510eba6b Add missing documentation for the PEP 293 functionality to
Walter Dörwald <walter@livinglogic.de>
parents: 26082
diff changeset
98 The .encode()/.decode() methods may use different error
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
99 handling schemes by providing the errors argument. These
26293
ff5b510eba6b Add missing documentation for the PEP 293 functionality to
Walter Dörwald <walter@livinglogic.de>
parents: 26082
diff changeset
100 string values are predefined:
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
101
12051
d89556c04ef9 Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12011
diff changeset
102 'strict' - raise a ValueError error (or a subclass)
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
103 'ignore' - ignore the character and continue with the next
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
104 'replace' - replace with a suitable replacement character;
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
105 Python will use the official U+FFFD REPLACEMENT
26293
ff5b510eba6b Add missing documentation for the PEP 293 functionality to
Walter Dörwald <walter@livinglogic.de>
parents: 26082
diff changeset
106 CHARACTER for the builtin Unicode codecs on
ff5b510eba6b Add missing documentation for the PEP 293 functionality to
Walter Dörwald <walter@livinglogic.de>
parents: 26082
diff changeset
107 decoding and '?' on encoding.
ff5b510eba6b Add missing documentation for the PEP 293 functionality to
Walter Dörwald <walter@livinglogic.de>
parents: 26082
diff changeset
108 'xmlcharrefreplace' - Replace with the appropriate XML
ff5b510eba6b Add missing documentation for the PEP 293 functionality to
Walter Dörwald <walter@livinglogic.de>
parents: 26082
diff changeset
109 character reference (only for encoding).
ff5b510eba6b Add missing documentation for the PEP 293 functionality to
Walter Dörwald <walter@livinglogic.de>
parents: 26082
diff changeset
110 'backslashreplace' - Replace with backslashed escape sequences
ff5b510eba6b Add missing documentation for the PEP 293 functionality to
Walter Dörwald <walter@livinglogic.de>
parents: 26082
diff changeset
111 (only for encoding).
ff5b510eba6b Add missing documentation for the PEP 293 functionality to
Walter Dörwald <walter@livinglogic.de>
parents: 26082
diff changeset
112
ff5b510eba6b Add missing documentation for the PEP 293 functionality to
Walter Dörwald <walter@livinglogic.de>
parents: 26082
diff changeset
113 The set of allowed values can be extended via register_error.
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
114
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
115 """
17839
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
116 def encode(self, input, errors='strict'):
12373
d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add
Guido van Rossum <guido@python.org>
parents: 12367
diff changeset
117
12010
bd1014215f62 "intput" --> "input" (in docstring); noted by Detlef Lannert
Fred Drake <fdrake@acm.org>
parents: 11990
diff changeset
118 """ Encodes the object input and returns a tuple (output
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
119 object, length consumed).
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
120
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
121 errors defines the error handling to apply. It defaults to
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
122 'strict' handling.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
123
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
124 The method may not store state in the Codec instance. Use
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
125 StreamCodec for codecs which have to keep state in order to
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
126 make encoding/decoding efficient.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
127
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
128 The encoder must be able to handle zero length input and
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
129 return an empty object of the output object type in this
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
130 situation.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
131
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
132 """
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
133 raise NotImplementedError
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
134
17839
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
135 def decode(self, input, errors='strict'):
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
136
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
137 """ Decodes the object input and returns a tuple (output
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
138 object, length consumed).
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
139
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
140 input must be an object which provides the bf_getreadbuf
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
141 buffer slot. Python strings, buffer objects and memory
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
142 mapped files are examples of objects providing this slot.
12373
d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add
Guido van Rossum <guido@python.org>
parents: 12367
diff changeset
143
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
144 errors defines the error handling to apply. It defaults to
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
145 'strict' handling.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
146
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
147 The method may not store state in the Codec instance. Use
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
148 StreamCodec for codecs which have to keep state in order to
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
149 make encoding/decoding efficient.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
150
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
151 The decoder must be able to handle zero length input and
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
152 return an empty object of the output object type in this
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
153 situation.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
154
12373
d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add
Guido van Rossum <guido@python.org>
parents: 12367
diff changeset
155 """
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
156 raise NotImplementedError
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
157
37875
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
158 class IncrementalEncoder(object):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
159 """
41473
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
160 An IncrementalEncoder encodes an input in multiple steps. The input can
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
161 be passed piece by piece to the encode() method. The IncrementalEncoder
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
162 remembers the state of the encoding process between calls to encode().
37875
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
163 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
164 def __init__(self, errors='strict'):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
165 """
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
166 Creates an IncrementalEncoder instance.
37875
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
167
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
168 The IncrementalEncoder may use different error handling schemes by
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
169 providing the errors keyword argument. See the module docstring
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
170 for a list of possible values.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
171 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
172 self.errors = errors
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
173 self.buffer = ""
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
174
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
175 def encode(self, input, final=False):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
176 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
177 Encodes input and returns the resulting object.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
178 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
179 raise NotImplementedError
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
180
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
181 def reset(self):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
182 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
183 Resets the encoder to the initial state.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
184 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
185
41473
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
186 def getstate(self):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
187 """
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
188 Return the current state of the encoder.
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
189 """
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
190 return 0
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
191
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
192 def setstate(self, state):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
193 """
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
194 Set the current state of the encoder. state must have been
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
195 returned by getstate().
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
196 """
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
197
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
198 class BufferedIncrementalEncoder(IncrementalEncoder):
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
199 """
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
200 This subclass of IncrementalEncoder can be used as the baseclass for an
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
201 incremental encoder if the encoder must keep some of the output in a
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
202 buffer between calls to encode().
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
203 """
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
204 def __init__(self, errors='strict'):
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
205 IncrementalEncoder.__init__(self, errors)
41473
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
206 # unencoded input that is kept between calls to encode()
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
207 self.buffer = ""
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
208
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
209 def _buffer_encode(self, input, errors, final):
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
210 # Overwrite this method in subclasses: It must encode input
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
211 # and return an (output, length consumed) tuple
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
212 raise NotImplementedError
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
213
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
214 def encode(self, input, final=False):
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
215 # encode input (taking the buffer into account)
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
216 data = self.buffer + input
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
217 (result, consumed) = self._buffer_encode(data, self.errors, final)
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
218 # keep unencoded input until the next call
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
219 self.buffer = data[consumed:]
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
220 return result
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
221
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
222 def reset(self):
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
223 IncrementalEncoder.reset(self)
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
224 self.buffer = ""
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 37875
diff changeset
225
41473
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
226 def getstate(self):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
227 return self.buffer or 0
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
228
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
229 def setstate(self, state):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
230 self.buffer = state or ""
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
231
37875
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
232 class IncrementalDecoder(object):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
233 """
41473
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
234 An IncrementalDecoder decodes an input in multiple steps. The input can
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
235 be passed piece by piece to the decode() method. The IncrementalDecoder
37875
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
236 remembers the state of the decoding process between calls to decode().
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
237 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
238 def __init__(self, errors='strict'):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
239 """
45835
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders;
Ka-Ping Yee <ping@zesty.ca>
parents: 45806
diff changeset
240 Create a IncrementalDecoder instance.
37875
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
241
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
242 The IncrementalDecoder may use different error handling schemes by
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
243 providing the errors keyword argument. See the module docstring
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
244 for a list of possible values.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
245 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
246 self.errors = errors
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
247
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
248 def decode(self, input, final=False):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
249 """
45835
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders;
Ka-Ping Yee <ping@zesty.ca>
parents: 45806
diff changeset
250 Decode input and returns the resulting object.
37875
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
251 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
252 raise NotImplementedError
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
253
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
254 def reset(self):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
255 """
45835
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders;
Ka-Ping Yee <ping@zesty.ca>
parents: 45806
diff changeset
256 Reset the decoder to the initial state.
37875
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
257 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
258
41473
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
259 def getstate(self):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
260 """
45835
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders;
Ka-Ping Yee <ping@zesty.ca>
parents: 45806
diff changeset
261 Return the current state of the decoder.
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders;
Ka-Ping Yee <ping@zesty.ca>
parents: 45806
diff changeset
262
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders;
Ka-Ping Yee <ping@zesty.ca>
parents: 45806
diff changeset
263 This must be a (buffered_input, additional_state_info) tuple.
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders;
Ka-Ping Yee <ping@zesty.ca>
parents: 45806
diff changeset
264 buffered_input must be a bytes object containing bytes that
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders;
Ka-Ping Yee <ping@zesty.ca>
parents: 45806
diff changeset
265 were passed to decode() that have not yet been converted.
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders;
Ka-Ping Yee <ping@zesty.ca>
parents: 45806
diff changeset
266 additional_state_info must be a non-negative integer
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders;
Ka-Ping Yee <ping@zesty.ca>
parents: 45806
diff changeset
267 representing the state of the decoder WITHOUT yet having
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders;
Ka-Ping Yee <ping@zesty.ca>
parents: 45806
diff changeset
268 processed the contents of buffered_input. In the initial state
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders;
Ka-Ping Yee <ping@zesty.ca>
parents: 45806
diff changeset
269 and after reset(), getstate() must return (b"", 0).
41473
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
270 """
41606
a3e3001b35bf Make the BOM constants in codecs.py bytes.
Walter Dörwald <walter@livinglogic.de>
parents: 41575
diff changeset
271 return (b"", 0)
41473
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
272
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
273 def setstate(self, state):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
274 """
45835
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders;
Ka-Ping Yee <ping@zesty.ca>
parents: 45806
diff changeset
275 Set the current state of the decoder.
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders;
Ka-Ping Yee <ping@zesty.ca>
parents: 45806
diff changeset
276
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders;
Ka-Ping Yee <ping@zesty.ca>
parents: 45806
diff changeset
277 state must have been returned by getstate(). The effect of
04050373d799 Make TextIOWrapper's seek/tell work properly with stateful decoders;
Ka-Ping Yee <ping@zesty.ca>
parents: 45806
diff changeset
278 setstate((b"", 0)) must be equivalent to reset().
41473
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
279 """
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
280
37875
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
281 class BufferedIncrementalDecoder(IncrementalDecoder):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
282 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
283 This subclass of IncrementalDecoder can be used as the baseclass for an
41473
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
284 incremental decoder if the decoder must be able to handle incomplete
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
285 byte sequences.
37875
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
286 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
287 def __init__(self, errors='strict'):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
288 IncrementalDecoder.__init__(self, errors)
41473
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
289 # undecoded input that is kept between calls to decode()
41606
a3e3001b35bf Make the BOM constants in codecs.py bytes.
Walter Dörwald <walter@livinglogic.de>
parents: 41575
diff changeset
290 self.buffer = b""
37875
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
291
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
292 def _buffer_decode(self, input, errors, final):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
293 # Overwrite this method in subclasses: It must decode input
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
294 # and return an (output, length consumed) tuple
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
295 raise NotImplementedError
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
296
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
297 def decode(self, input, final=False):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
298 # decode input (taking the buffer into account)
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
299 data = self.buffer + input
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
300 (result, consumed) = self._buffer_decode(data, self.errors, final)
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
301 # keep undecoded input until the next call
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
302 self.buffer = data[consumed:]
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
303 return result
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
304
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
305 def reset(self):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
306 IncrementalDecoder.reset(self)
41606
a3e3001b35bf Make the BOM constants in codecs.py bytes.
Walter Dörwald <walter@livinglogic.de>
parents: 41575
diff changeset
307 self.buffer = b""
37875
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
308
41473
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
309 def getstate(self):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
310 # additional state info is always 0
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
311 return (self.buffer, 0)
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
312
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
313 def setstate(self, state):
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
314 # ignore additional state info
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
315 self.buffer = state[0]
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
316
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
317 #
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
318 # The StreamWriter and StreamReader class provide generic working
19967
0b51e607f333 Fix typo in comment
Andrew M. Kuchling <amk@amk.ca>
parents: 18936
diff changeset
319 # interfaces which can be used to implement new encoding submodules
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
320 # very easily. See encodings/utf_8.py for an example on how this is
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
321 # done.
12373
d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add
Guido van Rossum <guido@python.org>
parents: 12367
diff changeset
322 #
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
323
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
324 class StreamWriter(Codec):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
325
17839
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
326 def __init__(self, stream, errors='strict'):
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
327
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
328 """ Creates a StreamWriter instance.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
329
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
330 stream must be a file-like object open for writing
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
331 (binary) data.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
332
26293
ff5b510eba6b Add missing documentation for the PEP 293 functionality to
Walter Dörwald <walter@livinglogic.de>
parents: 26082
diff changeset
333 The StreamWriter may use different error handling
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
334 schemes by providing the errors keyword argument. These
26293
ff5b510eba6b Add missing documentation for the PEP 293 functionality to
Walter Dörwald <walter@livinglogic.de>
parents: 26082
diff changeset
335 parameters are predefined:
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
336
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
337 'strict' - raise a ValueError (or a subclass)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
338 'ignore' - ignore the character and continue with the next
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
339 'replace'- replace with a suitable replacement character
26293
ff5b510eba6b Add missing documentation for the PEP 293 functionality to
Walter Dörwald <walter@livinglogic.de>
parents: 26082
diff changeset
340 'xmlcharrefreplace' - Replace with the appropriate XML
ff5b510eba6b Add missing documentation for the PEP 293 functionality to
Walter Dörwald <walter@livinglogic.de>
parents: 26082
diff changeset
341 character reference.
ff5b510eba6b Add missing documentation for the PEP 293 functionality to
Walter Dörwald <walter@livinglogic.de>
parents: 26082
diff changeset
342 'backslashreplace' - Replace with backslashed escape
ff5b510eba6b Add missing documentation for the PEP 293 functionality to
Walter Dörwald <walter@livinglogic.de>
parents: 26082
diff changeset
343 sequences (only for encoding).
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
344
26293
ff5b510eba6b Add missing documentation for the PEP 293 functionality to
Walter Dörwald <walter@livinglogic.de>
parents: 26082
diff changeset
345 The set of allowed parameter values can be extended via
ff5b510eba6b Add missing documentation for the PEP 293 functionality to
Walter Dörwald <walter@livinglogic.de>
parents: 26082
diff changeset
346 register_error.
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
347 """
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
348 self.stream = stream
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
349 self.errors = errors
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
350
12367
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
351 def write(self, object):
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
352
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
353 """ Writes the object's contents encoded to self.stream.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
354 """
17839
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
355 data, consumed = self.encode(object, self.errors)
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
356 self.stream.write(data)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
357
12367
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
358 def writelines(self, list):
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
359
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
360 """ Writes the concatenated list of strings to the stream
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
361 using .write().
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
362 """
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
363 self.write(''.join(list))
12373
d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add
Guido van Rossum <guido@python.org>
parents: 12367
diff changeset
364
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
365 def reset(self):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
366
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
367 """ Flushes and resets the codec buffers used for keeping state.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
368
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
369 Calling this method should ensure that the data on the
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
370 output is put into a clean state, that allows appending
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
371 of new fresh data without having to rescan the whole
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
372 stream to recover state.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
373
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
374 """
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
375 pass
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
376
61863
cd4936691163 Merged revisions 81474 via svnmerge from
Victor Stinner <victor.stinner@haypocalc.com>
parents: 61850
diff changeset
377 def seek(self, offset, whence=0):
cd4936691163 Merged revisions 81474 via svnmerge from
Victor Stinner <victor.stinner@haypocalc.com>
parents: 61850
diff changeset
378 self.stream.seek(offset, whence)
cd4936691163 Merged revisions 81474 via svnmerge from
Victor Stinner <victor.stinner@haypocalc.com>
parents: 61850
diff changeset
379 if whence == 0 and offset == 0:
cd4936691163 Merged revisions 81474 via svnmerge from
Victor Stinner <victor.stinner@haypocalc.com>
parents: 61850
diff changeset
380 self.reset()
cd4936691163 Merged revisions 81474 via svnmerge from
Victor Stinner <victor.stinner@haypocalc.com>
parents: 61850
diff changeset
381
17839
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
382 def __getattr__(self, name,
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
383 getattr=getattr):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
384
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
385 """ Inherit all other methods from the underlying stream.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
386 """
17839
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
387 return getattr(self.stream, name)
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
388
40450
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
389 def __enter__(self):
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
390 return self
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
391
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
392 def __exit__(self, type, value, tb):
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
393 self.stream.close()
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
394
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
395 ###
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
396
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
397 class StreamReader(Codec):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
398
17839
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
399 def __init__(self, stream, errors='strict'):
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
400
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
401 """ Creates a StreamReader instance.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
402
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
403 stream must be a file-like object open for reading
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
404 (binary) data.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
405
26293
ff5b510eba6b Add missing documentation for the PEP 293 functionality to
Walter Dörwald <walter@livinglogic.de>
parents: 26082
diff changeset
406 The StreamReader may use different error handling
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
407 schemes by providing the errors keyword argument. These
26293
ff5b510eba6b Add missing documentation for the PEP 293 functionality to
Walter Dörwald <walter@livinglogic.de>
parents: 26082
diff changeset
408 parameters are predefined:
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
409
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
410 'strict' - raise a ValueError (or a subclass)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
411 'ignore' - ignore the character and continue with the next
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
412 'replace'- replace with a suitable replacement character;
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
413
26293
ff5b510eba6b Add missing documentation for the PEP 293 functionality to
Walter Dörwald <walter@livinglogic.de>
parents: 26082
diff changeset
414 The set of allowed parameter values can be extended via
ff5b510eba6b Add missing documentation for the PEP 293 functionality to
Walter Dörwald <walter@livinglogic.de>
parents: 26082
diff changeset
415 register_error.
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
416 """
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
417 self.stream = stream
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
418 self.errors = errors
41606
a3e3001b35bf Make the BOM constants in codecs.py bytes.
Walter Dörwald <walter@livinglogic.de>
parents: 41575
diff changeset
419 self.bytebuffer = b""
35269
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead
Walter Dörwald <walter@livinglogic.de>
parents: 34924
diff changeset
420 # For str->str decoding this will stay a str
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead
Walter Dörwald <walter@livinglogic.de>
parents: 34924
diff changeset
421 # For str->unicode decoding the first read will promote it to unicode
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead
Walter Dörwald <walter@livinglogic.de>
parents: 34924
diff changeset
422 self.charbuffer = ""
35589
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
423 self.linebuffer = None
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
424
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
425 def decode(self, input, errors='strict'):
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
426 raise NotImplementedError
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
427
35415
ff2e66fc0a29 Return complete lines from codec stream readers
Martin v. Löwis <martin@v.loewis.de>
parents: 35269
diff changeset
428 def read(self, size=-1, chars=-1, firstline=False):
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
429
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
430 """ Decodes data from the stream self.stream and returns the
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
431 resulting object.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
432
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
433 chars indicates the number of characters to read from the
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
434 stream. read() will never return more than chars
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
435 characters, but it might return less, if there are not enough
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
436 characters available.
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
437
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
438 size indicates the approximate maximum number of bytes to
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
439 read from the stream for decoding purposes. The decoder
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
440 can modify this setting as appropriate. The default value
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
441 -1 indicates to read and decode as much as possible. size
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
442 is intended to prevent having to decode huge files in one
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
443 step.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
444
35415
ff2e66fc0a29 Return complete lines from codec stream readers
Martin v. Löwis <martin@v.loewis.de>
parents: 35269
diff changeset
445 If firstline is true, and a UnicodeDecodeError happens
ff2e66fc0a29 Return complete lines from codec stream readers
Martin v. Löwis <martin@v.loewis.de>
parents: 35269
diff changeset
446 after the first line terminator in the input only the first line
ff2e66fc0a29 Return complete lines from codec stream readers
Martin v. Löwis <martin@v.loewis.de>
parents: 35269
diff changeset
447 will be returned, the rest of the input will be kept until the
ff2e66fc0a29 Return complete lines from codec stream readers
Martin v. Löwis <martin@v.loewis.de>
parents: 35269
diff changeset
448 next call to read().
ff2e66fc0a29 Return complete lines from codec stream readers
Martin v. Löwis <martin@v.loewis.de>
parents: 35269
diff changeset
449
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
450 The method should use a greedy read strategy meaning that
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
451 it should read as much data as is allowed within the
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
452 definition of the encoding and the given size, e.g. if
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
453 optional encoding endings or state markers are available
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
454 on the stream, these should be read too.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
455 """
35589
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
456 # If we have lines cached, first merge them back into characters
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
457 if self.linebuffer:
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
458 self.charbuffer = "".join(self.linebuffer)
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
459 self.linebuffer = None
36121
686306a67e9b Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 35589
diff changeset
460
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
461 # read until we get the required number of characters (if available)
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
462 while True:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
463 # can the request can be satisfied from the character buffer?
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
464 if chars < 0:
36855
204c307b01ce If size is specified, try to read at least size characters.
Walter Dörwald <walter@livinglogic.de>
parents: 36121
diff changeset
465 if size < 0:
204c307b01ce If size is specified, try to read at least size characters.
Walter Dörwald <walter@livinglogic.de>
parents: 36121
diff changeset
466 if self.charbuffer:
204c307b01ce If size is specified, try to read at least size characters.
Walter Dörwald <walter@livinglogic.de>
parents: 36121
diff changeset
467 break
204c307b01ce If size is specified, try to read at least size characters.
Walter Dörwald <walter@livinglogic.de>
parents: 36121
diff changeset
468 elif len(self.charbuffer) >= size:
34214
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
469 break
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
470 else:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
471 if len(self.charbuffer) >= chars:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
472 break
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
473 # we need more data
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
474 if size < 0:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
475 newdata = self.stream.read()
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
476 else:
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
477 newdata = self.stream.read(size)
34214
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
478 # decode bytes (those remaining from the last call included)
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
479 data = self.bytebuffer + newdata
35415
ff2e66fc0a29 Return complete lines from codec stream readers
Martin v. Löwis <martin@v.loewis.de>
parents: 35269
diff changeset
480 try:
ff2e66fc0a29 Return complete lines from codec stream readers
Martin v. Löwis <martin@v.loewis.de>
parents: 35269
diff changeset
481 newchars, decodedbytes = self.decode(data, self.errors)
40587
6b18a095c2e7 SF patch 1631942 by Collin Winter:
Guido van Rossum <guido@python.org>
parents: 40450
diff changeset
482 except UnicodeDecodeError as exc:
35415
ff2e66fc0a29 Return complete lines from codec stream readers
Martin v. Löwis <martin@v.loewis.de>
parents: 35269
diff changeset
483 if firstline:
41473
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
484 newchars, decodedbytes = \
a49859f44c43 Apply SF patch #1698994: Add getstate() and setstate()
Walter Dörwald <walter@livinglogic.de>
parents: 40587
diff changeset
485 self.decode(data[:exc.start], self.errors)
35415
ff2e66fc0a29 Return complete lines from codec stream readers
Martin v. Löwis <martin@v.loewis.de>
parents: 35269
diff changeset
486 lines = newchars.splitlines(True)
ff2e66fc0a29 Return complete lines from codec stream readers
Martin v. Löwis <martin@v.loewis.de>
parents: 35269
diff changeset
487 if len(lines)<=1:
ff2e66fc0a29 Return complete lines from codec stream readers
Martin v. Löwis <martin@v.loewis.de>
parents: 35269
diff changeset
488 raise
ff2e66fc0a29 Return complete lines from codec stream readers
Martin v. Löwis <martin@v.loewis.de>
parents: 35269
diff changeset
489 else:
ff2e66fc0a29 Return complete lines from codec stream readers
Martin v. Löwis <martin@v.loewis.de>
parents: 35269
diff changeset
490 raise
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
491 # keep undecoded bytes until the next call
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
492 self.bytebuffer = data[decodedbytes:]
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
493 # put new characters in the character buffer
34214
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
494 self.charbuffer += newchars
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
495 # there was no data available
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
496 if not newdata:
34214
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
497 break
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
498 if chars < 0:
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
499 # Return everything we've got
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
500 result = self.charbuffer
35269
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead
Walter Dörwald <walter@livinglogic.de>
parents: 34924
diff changeset
501 self.charbuffer = ""
34214
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
502 else:
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
503 # Return the first chars characters
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
504 result = self.charbuffer[:chars]
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
505 self.charbuffer = self.charbuffer[chars:]
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
506 return result
12373
d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add
Guido van Rossum <guido@python.org>
parents: 12367
diff changeset
507
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
508 def readline(self, size=None, keepends=True):
12367
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
509
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
510 """ Read one line from the input stream and return the
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
511 decoded data.
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
512
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
513 size, if given, is passed as size argument to the
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
514 read() method.
12373
d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add
Guido van Rossum <guido@python.org>
parents: 12367
diff changeset
515
12367
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
516 """
35589
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
517 # If we have lines cached from an earlier read, return
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
518 # them unconditionally
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
519 if self.linebuffer:
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
520 line = self.linebuffer[0]
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
521 del self.linebuffer[0]
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
522 if len(self.linebuffer) == 1:
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
523 # revert to charbuffer mode; we might need more data
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
524 # next time
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
525 self.charbuffer = self.linebuffer[0]
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
526 self.linebuffer = None
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
527 if not keepends:
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
528 line = line.splitlines(False)[0]
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
529 return line
36121
686306a67e9b Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 35589
diff changeset
530
34214
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
531 readsize = size or 72
35269
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead
Walter Dörwald <walter@livinglogic.de>
parents: 34924
diff changeset
532 line = ""
34214
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
533 # If size is given, we call read() only once
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
534 while True:
35415
ff2e66fc0a29 Return complete lines from codec stream readers
Martin v. Löwis <martin@v.loewis.de>
parents: 35269
diff changeset
535 data = self.read(readsize, firstline=True)
34214
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
536 if data:
34924
c66a7fbe16cf Fix comment.
Walter Dörwald <walter@livinglogic.de>
parents: 34923
diff changeset
537 # If we're at a "\r" read one extra character (which might
c66a7fbe16cf Fix comment.
Walter Dörwald <walter@livinglogic.de>
parents: 34923
diff changeset
538 # be a "\n") to get a proper line ending. If the stream is
34923
bc123b9cef7f If the data read from the bytestream in readline() ends in a '\r' read one more
Walter Dörwald <walter@livinglogic.de>
parents: 34866
diff changeset
539 # temporarily exhausted we return the wrong line ending.
35269
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead
Walter Dörwald <walter@livinglogic.de>
parents: 34924
diff changeset
540 if data.endswith("\r"):
34865
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if
Walter Dörwald <walter@livinglogic.de>
parents: 34761
diff changeset
541 data += self.read(size=1, chars=1)
b49d39111c48 Fix for SF bug #1175396: readline() will now read one more character, if
Walter Dörwald <walter@livinglogic.de>
parents: 34761
diff changeset
542
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
543 line += data
34214
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
544 lines = line.splitlines(True)
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
545 if lines:
35589
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
546 if len(lines) > 1:
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
547 # More than one line result; the first line is a full line
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
548 # to return
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
549 line = lines[0]
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
550 del lines[0]
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
551 if len(lines) > 1:
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
552 # cache the remaining lines
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
553 lines[-1] += self.charbuffer
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
554 self.linebuffer = lines
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
555 self.charbuffer = None
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
556 else:
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
557 # only one remaining line, put it back into charbuffer
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
558 self.charbuffer = lines[0] + self.charbuffer
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
559 if not keepends:
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
560 line = line.splitlines(False)[0]
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
561 break
34214
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
562 line0withend = lines[0]
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
563 line0withoutend = lines[0].splitlines(False)[0]
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
564 if line0withend != line0withoutend: # We really have a line end
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
565 # Put the rest back together and keep it until the next call
35269
2b117eb18944 Make attributes and local variables in the StreamReader str objects instead
Walter Dörwald <walter@livinglogic.de>
parents: 34924
diff changeset
566 self.charbuffer = "".join(lines[1:]) + self.charbuffer
34214
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
567 if keepends:
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
568 line = line0withend
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
569 else:
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
570 line = line0withoutend
34387
f25dcd7f3e48 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34214
diff changeset
571 break
34214
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
572 # we didn't get anything or this was our only try
34387
f25dcd7f3e48 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34214
diff changeset
573 if not data or size is not None:
34214
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
574 if line and not keepends:
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
575 line = line.splitlines(False)[0]
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
576 break
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
577 if readsize<8000:
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
578 readsize *= 2
abe3145dde4e Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 33782
diff changeset
579 return line
12373
d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add
Guido van Rossum <guido@python.org>
parents: 12367
diff changeset
580
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
581 def readlines(self, sizehint=None, keepends=True):
12367
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
582
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
583 """ Read all lines available on the input stream
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
584 and return them as list of lines.
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
585
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
586 Line breaks are implemented using the codec's decoder
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
587 method and are included in the list entries.
12373
d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add
Guido van Rossum <guido@python.org>
parents: 12367
diff changeset
588
31591
26a80f0af9f3 Ignore sizehint argument. Fixes SF #844561.
Marc-André Lemburg <mal@egenix.com>
parents: 27618
diff changeset
589 sizehint, if given, is ignored since there is no efficient
26a80f0af9f3 Ignore sizehint argument. Fixes SF #844561.
Marc-André Lemburg <mal@egenix.com>
parents: 27618
diff changeset
590 way to finding the true end-of-line.
12367
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
591
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
592 """
33428
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support
Walter Dörwald <walter@livinglogic.de>
parents: 31591
diff changeset
593 data = self.read()
33782
a52de911b284 SF #1048865: Fix a trivial typo that breaks StreamReader.readlines()
Hye-Shik Chang <hyeshik@gmail.com>
parents: 33428
diff changeset
594 return data.splitlines(keepends)
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
595
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
596 def reset(self):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
597
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
598 """ Resets the codec buffers used for keeping state.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
599
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
600 Note that no stream repositioning should take place.
13620
5b727ab37092 Spelling fixes supplied by Rob W. W. Hooft. All these are fixes in either
Thomas Wouters <thomas@python.org>
parents: 12986
diff changeset
601 This method is primarily intended to be able to recover
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
602 from decoding errors.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
603
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
604 """
41606
a3e3001b35bf Make the BOM constants in codecs.py bytes.
Walter Dörwald <walter@livinglogic.de>
parents: 41575
diff changeset
605 self.bytebuffer = b""
41575
beacf40b5960 Rip out all the u"..." literals and calls to unicode().
Guido van Rossum <guido@python.org>
parents: 41501
diff changeset
606 self.charbuffer = ""
35589
5f2785722cba Patch #1268314: Cache lines in StreamReader.readlines for performance.
Martin v. Löwis <martin@v.loewis.de>
parents: 35508
diff changeset
607 self.linebuffer = None
34751
962800ee30ef Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34707
diff changeset
608
34753
11b4350b5879 Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34751
diff changeset
609 def seek(self, offset, whence=0):
34751
962800ee30ef Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34707
diff changeset
610 """ Set the input stream's current position.
962800ee30ef Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34707
diff changeset
611
962800ee30ef Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34707
diff changeset
612 Resets the codec buffers used for keeping state.
962800ee30ef Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34707
diff changeset
613 """
61863
cd4936691163 Merged revisions 81474 via svnmerge from
Victor Stinner <victor.stinner@haypocalc.com>
parents: 61850
diff changeset
614 self.stream.seek(offset, whence)
34751
962800ee30ef Backport checkin:
Walter Dörwald <walter@livinglogic.de>
parents: 34707
diff changeset
615 self.reset()
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
616
41501
0c5019afc55b PEP 3114: rename .next() to .__next__() and add next() builtin.
Georg Brandl <georg@python.org>
parents: 41473
diff changeset
617 def __next__(self):
26082
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter
Walter Dörwald <walter@livinglogic.de>
parents: 25091
diff changeset
618
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter
Walter Dörwald <walter@livinglogic.de>
parents: 25091
diff changeset
619 """ Return the next decoded line from the input stream."""
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter
Walter Dörwald <walter@livinglogic.de>
parents: 25091
diff changeset
620 line = self.readline()
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter
Walter Dörwald <walter@livinglogic.de>
parents: 25091
diff changeset
621 if line:
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter
Walter Dörwald <walter@livinglogic.de>
parents: 25091
diff changeset
622 return line
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter
Walter Dörwald <walter@livinglogic.de>
parents: 25091
diff changeset
623 raise StopIteration
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter
Walter Dörwald <walter@livinglogic.de>
parents: 25091
diff changeset
624
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter
Walter Dörwald <walter@livinglogic.de>
parents: 25091
diff changeset
625 def __iter__(self):
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter
Walter Dörwald <walter@livinglogic.de>
parents: 25091
diff changeset
626 return self
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter
Walter Dörwald <walter@livinglogic.de>
parents: 25091
diff changeset
627
17839
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
628 def __getattr__(self, name,
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
629 getattr=getattr):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
630
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
631 """ Inherit all other methods from the underlying stream.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
632 """
17839
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
633 return getattr(self.stream, name)
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
634
40450
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
635 def __enter__(self):
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
636 return self
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
637
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
638 def __exit__(self, type, value, tb):
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
639 self.stream.close()
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
640
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
641 ###
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
642
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
643 class StreamReaderWriter:
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
644
12389
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
645 """ StreamReaderWriter instances allow wrapping streams which
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
646 work in both read and write modes.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
647
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
648 The design is such that one can use the factory functions
13620
5b727ab37092 Spelling fixes supplied by Rob W. W. Hooft. All these are fixes in either
Thomas Wouters <thomas@python.org>
parents: 12986
diff changeset
649 returned by the codec.lookup() function to construct the
12389
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
650 instance.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
651
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
652 """
12367
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
653 # Optional attributes set by the file wrappers below
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
654 encoding = 'unknown'
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
655
17839
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
656 def __init__(self, stream, Reader, Writer, errors='strict'):
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
657
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
658 """ Creates a StreamReaderWriter instance.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
659
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
660 stream must be a Stream-like object.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
661
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
662 Reader, Writer must be factory functions or classes
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
663 providing the StreamReader, StreamWriter interface resp.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
664
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
665 Error handling is done in the same way as defined for the
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
666 StreamWriter/Readers.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
667
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
668 """
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
669 self.stream = stream
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
670 self.reader = Reader(stream, errors)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
671 self.writer = Writer(stream, errors)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
672 self.errors = errors
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
673
17839
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
674 def read(self, size=-1):
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
675
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
676 return self.reader.read(size)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
677
12534
189298a1fab4 Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12389
diff changeset
678 def readline(self, size=None):
12367
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
679
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
680 return self.reader.readline(size)
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
681
12534
189298a1fab4 Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12389
diff changeset
682 def readlines(self, sizehint=None):
12367
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
683
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
684 return self.reader.readlines(sizehint)
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
685
41501
0c5019afc55b PEP 3114: rename .next() to .__next__() and add next() builtin.
Georg Brandl <georg@python.org>
parents: 41473
diff changeset
686 def __next__(self):
26082
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter
Walter Dörwald <walter@livinglogic.de>
parents: 25091
diff changeset
687
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter
Walter Dörwald <walter@livinglogic.de>
parents: 25091
diff changeset
688 """ Return the next decoded line from the input stream."""
41501
0c5019afc55b PEP 3114: rename .next() to .__next__() and add next() builtin.
Georg Brandl <georg@python.org>
parents: 41473
diff changeset
689 return next(self.reader)
26082
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter
Walter Dörwald <walter@livinglogic.de>
parents: 25091
diff changeset
690
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter
Walter Dörwald <walter@livinglogic.de>
parents: 25091
diff changeset
691 def __iter__(self):
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter
Walter Dörwald <walter@livinglogic.de>
parents: 25091
diff changeset
692 return self
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter
Walter Dörwald <walter@livinglogic.de>
parents: 25091
diff changeset
693
17839
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
694 def write(self, data):
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
695
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
696 return self.writer.write(data)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
697
17839
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
698 def writelines(self, list):
12367
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
699
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
700 return self.writer.writelines(list)
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
701
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
702 def reset(self):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
703
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
704 self.reader.reset()
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
705 self.writer.reset()
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
706
61850
846dbcf970ab Merged revisions 81461 via svnmerge from
Victor Stinner <victor.stinner@haypocalc.com>
parents: 45835
diff changeset
707 def seek(self, offset, whence=0):
61863
cd4936691163 Merged revisions 81474 via svnmerge from
Victor Stinner <victor.stinner@haypocalc.com>
parents: 61850
diff changeset
708 self.stream.seek(offset, whence)
cd4936691163 Merged revisions 81474 via svnmerge from
Victor Stinner <victor.stinner@haypocalc.com>
parents: 61850
diff changeset
709 self.reader.reset()
cd4936691163 Merged revisions 81474 via svnmerge from
Victor Stinner <victor.stinner@haypocalc.com>
parents: 61850
diff changeset
710 if whence == 0 and offset == 0:
cd4936691163 Merged revisions 81474 via svnmerge from
Victor Stinner <victor.stinner@haypocalc.com>
parents: 61850
diff changeset
711 self.writer.reset()
61850
846dbcf970ab Merged revisions 81461 via svnmerge from
Victor Stinner <victor.stinner@haypocalc.com>
parents: 45835
diff changeset
712
17839
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
713 def __getattr__(self, name,
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
714 getattr=getattr):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
715
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
716 """ Inherit all other methods from the underlying stream.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
717 """
17839
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
718 return getattr(self.stream, name)
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
719
40450
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
720 # these are needed to make "with codecs.open(...)" work properly
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
721
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
722 def __enter__(self):
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
723 return self
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
724
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
725 def __exit__(self, type, value, tb):
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
726 self.stream.close()
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
727
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
728 ###
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
729
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
730 class StreamRecoder:
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
731
12389
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
732 """ StreamRecoder instances provide a frontend - backend
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
733 view of encoding data.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
734
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
735 They use the complete set of APIs returned by the
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
736 codecs.lookup() function to implement their task.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
737
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
738 Data written to the stream is first decoded into an
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
739 intermediate format (which is dependent on the given codec
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
740 combination) and then written to the stream using an instance
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
741 of the provided Writer class.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
742
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
743 In the other direction, data is read from the stream using a
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
744 Reader instance and then return encoded data to the caller.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
745
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
746 """
12367
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
747 # Optional attributes set by the file wrappers below
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
748 data_encoding = 'unknown'
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
749 file_encoding = 'unknown'
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
750
17839
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
751 def __init__(self, stream, encode, decode, Reader, Writer,
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
752 errors='strict'):
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
753
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
754 """ Creates a StreamRecoder instance which implements a two-way
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
755 conversion: encode and decode work on the frontend (the
12373
d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add
Guido van Rossum <guido@python.org>
parents: 12367
diff changeset
756 input to .read() and output of .write()) while
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
757 Reader and Writer work on the backend (reading and
12011
4fa25a292273 Oops, another in the same file; I should read the mail fully before
Fred Drake <fdrake@acm.org>
parents: 12010
diff changeset
758 writing to the stream).
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
759
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
760 You can use these objects to do transparent direct
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
761 recodings from e.g. latin-1 to utf-8 and back.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
762
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
763 stream must be a file-like object.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
764
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
765 encode, decode must adhere to the Codec interface, Reader,
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
766 Writer must be factory functions or classes providing the
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
767 StreamReader, StreamWriter interface resp.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
768
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
769 encode and decode are needed for the frontend translation,
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
770 Reader and Writer for the backend translation. Unicode is
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
771 used as intermediate encoding.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
772
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
773 Error handling is done in the same way as defined for the
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
774 StreamWriter/Readers.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
775
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
776 """
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
777 self.stream = stream
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
778 self.encode = encode
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
779 self.decode = decode
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
780 self.reader = Reader(stream, errors)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
781 self.writer = Writer(stream, errors)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
782 self.errors = errors
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
783
17839
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
784 def read(self, size=-1):
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
785
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
786 data = self.reader.read(size)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
787 data, bytesencoded = self.encode(data, self.errors)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
788 return data
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
789
17839
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
790 def readline(self, size=None):
12367
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
791
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
792 if size is None:
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
793 data = self.reader.readline()
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
794 else:
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
795 data = self.reader.readline(size)
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
796 data, bytesencoded = self.encode(data, self.errors)
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
797 return data
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
798
17839
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
799 def readlines(self, sizehint=None):
12367
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
800
31591
26a80f0af9f3 Ignore sizehint argument. Fixes SF #844561.
Marc-André Lemburg <mal@egenix.com>
parents: 27618
diff changeset
801 data = self.reader.read()
12367
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
802 data, bytesencoded = self.encode(data, self.errors)
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
803 return data.splitlines(1)
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
804
41501
0c5019afc55b PEP 3114: rename .next() to .__next__() and add next() builtin.
Georg Brandl <georg@python.org>
parents: 41473
diff changeset
805 def __next__(self):
26082
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter
Walter Dörwald <walter@livinglogic.de>
parents: 25091
diff changeset
806
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter
Walter Dörwald <walter@livinglogic.de>
parents: 25091
diff changeset
807 """ Return the next decoded line from the input stream."""
41501
0c5019afc55b PEP 3114: rename .next() to .__next__() and add next() builtin.
Georg Brandl <georg@python.org>
parents: 41473
diff changeset
808 data = next(self.reader)
35508
1f2af259968b SF bug #1235646: codecs.StreamRecoder.next() now reencodes the data it reads
Walter Dörwald <walter@livinglogic.de>
parents: 35415
diff changeset
809 data, bytesencoded = self.encode(data, self.errors)
1f2af259968b SF bug #1235646: codecs.StreamRecoder.next() now reencodes the data it reads
Walter Dörwald <walter@livinglogic.de>
parents: 35415
diff changeset
810 return data
26082
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter
Walter Dörwald <walter@livinglogic.de>
parents: 25091
diff changeset
811
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter
Walter Dörwald <walter@livinglogic.de>
parents: 25091
diff changeset
812 def __iter__(self):
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter
Walter Dörwald <walter@livinglogic.de>
parents: 25091
diff changeset
813 return self
f3a64035ba8b Add next() and __iter__() methods to StreamReader, StreamReaderWriter
Walter Dörwald <walter@livinglogic.de>
parents: 25091
diff changeset
814
17839
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
815 def write(self, data):
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
816
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
817 data, bytesdecoded = self.decode(data, self.errors)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
818 return self.writer.write(data)
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
819
17839
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
820 def writelines(self, list):
12367
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
821
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
822 data = ''.join(list)
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
823 data, bytesdecoded = self.decode(data, self.errors)
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
824 return self.writer.write(data)
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
825
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
826 def reset(self):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
827
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
828 self.reader.reset()
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
829 self.writer.reset()
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
830
17839
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
831 def __getattr__(self, name,
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
832 getattr=getattr):
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
833
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
834 """ Inherit all other methods from the underlying stream.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
835 """
17839
c872fdf10130 Just changed "x,y" to "x, y" everywhere (i.e., inserted horizontal space
Tim Peters <tim.peters@gmail.com>
parents: 16070
diff changeset
836 return getattr(self.stream, name)
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
837
40450
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
838 def __enter__(self):
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
839 return self
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
840
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
841 def __exit__(self, type, value, tb):
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
842 self.stream.close()
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
843
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
844 ### Shortcuts
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
845
12986
5279a6e00d36 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12534
diff changeset
846 def open(filename, mode='rb', encoding=None, errors='strict', buffering=1):
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
847
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
848 """ Open an encoded file using the given mode and return
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
849 a wrapped version providing transparent encoding/decoding.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
850
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
851 Note: The wrapped version will only accept the object format
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
852 defined by the codecs, i.e. Unicode objects for most builtin
34761
Skip Montanaro <skip@pobox.com>
parents: 34753
diff changeset
853 codecs. Output is also codec dependent and will usually be
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
854 Unicode as well.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
855
12986
5279a6e00d36 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12534
diff changeset
856 Files are always opened in binary mode, even if no binary mode
27618
74d9604e9353 Fix typos.
Walter Dörwald <walter@livinglogic.de>
parents: 26906
diff changeset
857 was specified. This is done to avoid data loss due to encodings
12986
5279a6e00d36 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12534
diff changeset
858 using 8-bit values. The default file mode is 'rb' meaning to
5279a6e00d36 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12534
diff changeset
859 open the file in binary read mode.
5279a6e00d36 Marc-Andre Lemburg <mal@lemburg.com>:
Marc-André Lemburg <mal@egenix.com>
parents: 12534
diff changeset
860
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
861 encoding specifies the encoding which is to be used for the
27618
74d9604e9353 Fix typos.
Walter Dörwald <walter@livinglogic.de>
parents: 26906
diff changeset
862 file.
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
863
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
864 errors may be given to define the error handling. It defaults
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
865 to 'strict' which causes ValueErrors to be raised in case an
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
866 encoding error occurs.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
867
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
868 buffering has the same meaning as for the builtin open() API.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
869 It defaults to line buffered.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
870
12389
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
871 The returned wrapped file object provides an extra attribute
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
872 .encoding which allows querying the used encoding. This
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
873 attribute is only available if an encoding was specified as
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
874 parameter.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
875
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
876 """
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
877 if encoding is not None and \
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
878 'b' not in mode:
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
879 # Force opening of the file in binary mode
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
880 mode = mode + 'b'
44098
0889daf6d0d9 #1535: rename __builtin__ module to builtins.
Georg Brandl <georg@python.org>
parents: 41606
diff changeset
881 file = builtins.open(filename, mode, buffering)
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
882 if encoding is None:
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
883 return file
37875
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
884 info = lookup(encoding)
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
885 srw = StreamReaderWriter(file, info.streamreader, info.streamwriter, errors)
12367
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
886 # Add attributes to simplify introspection
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
887 srw.encoding = encoding
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
888 return srw
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
889
12367
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
890 def EncodedFile(file, data_encoding, file_encoding=None, errors='strict'):
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
891
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
892 """ Return a wrapped version of file which provides transparent
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
893 encoding translation.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
894
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
895 Strings written to the wrapped file are interpreted according
12367
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
896 to the given data_encoding and then written to the original
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
897 file as string using file_encoding. The intermediate encoding
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
898 will usually be Unicode but depends on the specified codecs.
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
899
12367
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
900 Strings are read from the file using file_encoding and then
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
901 passed back to the caller as string using data_encoding.
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
902
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
903 If file_encoding is not given, it defaults to data_encoding.
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
904
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
905 errors may be given to define the error handling. It defaults
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
906 to 'strict' which causes ValueErrors to be raised in case an
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
907 encoding error occurs.
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
908
12389
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
909 The returned wrapped file object provides two extra attributes
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
910 .data_encoding and .file_encoding which reflect the given
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
911 parameters of the same name. The attributes can be used for
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
912 introspection by Python programs.
e5b0bea03cfa M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12373
diff changeset
913
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
914 """
12367
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
915 if file_encoding is None:
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
916 file_encoding = data_encoding
40450
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
917 data_info = lookup(data_encoding)
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
918 file_info = lookup(file_encoding)
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
919 sr = StreamRecoder(file, data_info.encode, data_info.decode,
cc992d75d5b3 Four months of trunk changes (including a few releases...)
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
920 file_info.streamreader, file_info.streamwriter, errors)
12367
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
921 # Add attributes to simplify introspection
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
922 sr.data_encoding = data_encoding
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
923 sr.file_encoding = file_encoding
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
924 return sr
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
925
19981
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
926 ### Helpers for codec lookup
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
927
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
928 def getencoder(encoding):
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
929
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
930 """ Lookup up the codec for the given encoding and return
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
931 its encoder function.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
932
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
933 Raises a LookupError in case the encoding cannot be found.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
934
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
935 """
37875
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
936 return lookup(encoding).encode
19981
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
937
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
938 def getdecoder(encoding):
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
939
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
940 """ Lookup up the codec for the given encoding and return
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
941 its decoder function.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
942
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
943 Raises a LookupError in case the encoding cannot be found.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
944
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
945 """
37875
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
946 return lookup(encoding).decode
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
947
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
948 def getincrementalencoder(encoding):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
949
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
950 """ Lookup up the codec for the given encoding and return
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
951 its IncrementalEncoder class or factory function.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
952
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
953 Raises a LookupError in case the encoding cannot be found
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
954 or the codecs doesn't provide an incremental encoder.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
955
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
956 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
957 encoder = lookup(encoding).incrementalencoder
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
958 if encoder is None:
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
959 raise LookupError(encoding)
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
960 return encoder
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
961
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
962 def getincrementaldecoder(encoding):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
963
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
964 """ Lookup up the codec for the given encoding and return
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
965 its IncrementalDecoder class or factory function.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
966
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
967 Raises a LookupError in case the encoding cannot be found
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
968 or the codecs doesn't provide an incremental decoder.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
969
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
970 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
971 decoder = lookup(encoding).incrementaldecoder
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
972 if decoder is None:
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
973 raise LookupError(encoding)
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
974 return decoder
19981
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
975
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
976 def getreader(encoding):
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
977
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
978 """ Lookup up the codec for the given encoding and return
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
979 its StreamReader class or factory function.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
980
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
981 Raises a LookupError in case the encoding cannot be found.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
982
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
983 """
37875
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
984 return lookup(encoding).streamreader
19981
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
985
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
986 def getwriter(encoding):
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
987
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
988 """ Lookup up the codec for the given encoding and return
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
989 its StreamWriter class or factory function.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
990
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
991 Raises a LookupError in case the encoding cannot be found.
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
992
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
993 """
37875
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
994 return lookup(encoding).streamwriter
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
995
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
996 def iterencode(iterator, encoding, errors='strict', **kwargs):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
997 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
998 Encoding iterator.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
999
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1000 Encodes the input strings from the iterator using a IncrementalEncoder.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1001
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1002 errors and kwargs are passed through to the IncrementalEncoder
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1003 constructor.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1004 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1005 encoder = getincrementalencoder(encoding)(errors, **kwargs)
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1006 for input in iterator:
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1007 output = encoder.encode(input)
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1008 if output:
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1009 yield output
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1010 output = encoder.encode("", True)
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1011 if output:
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1012 yield output
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1013
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1014 def iterdecode(iterator, encoding, errors='strict', **kwargs):
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1015 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1016 Decoding iterator.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1017
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1018 Decodes the input strings from the iterator using a IncrementalDecoder.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1019
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1020 errors and kwargs are passed through to the IncrementalDecoder
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1021 constructor.
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1022 """
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1023 decoder = getincrementaldecoder(encoding)(errors, **kwargs)
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1024 for input in iterator:
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1025 output = decoder.decode(input)
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1026 if output:
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1027 yield output
41606
a3e3001b35bf Make the BOM constants in codecs.py bytes.
Walter Dörwald <walter@livinglogic.de>
parents: 41575
diff changeset
1028 output = decoder.decode(b"", True)
37875
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1029 if output:
a4c70f3968be Merge part of the trunk changes into the p3yk branch. This merges from 43030
Thomas Wouters <thomas@python.org>
parents: 36855
diff changeset
1030 yield output
19981
0fd95200ab6e Added new helpers for easy access to codecs. Docs will follow.
Marc-André Lemburg <mal@egenix.com>
parents: 19967
diff changeset
1031
15706
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap
Marc-André Lemburg <mal@egenix.com>
parents: 15498
diff changeset
1032 ### Helpers for charmap-based codecs
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap
Marc-André Lemburg <mal@egenix.com>
parents: 15498
diff changeset
1033
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap
Marc-André Lemburg <mal@egenix.com>
parents: 15498
diff changeset
1034 def make_identity_dict(rng):
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap
Marc-André Lemburg <mal@egenix.com>
parents: 15498
diff changeset
1035
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap
Marc-André Lemburg <mal@egenix.com>
parents: 15498
diff changeset
1036 """ make_identity_dict(rng) -> dict
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap
Marc-André Lemburg <mal@egenix.com>
parents: 15498
diff changeset
1037
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap
Marc-André Lemburg <mal@egenix.com>
parents: 15498
diff changeset
1038 Return a dictionary where elements of the rng sequence are
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap
Marc-André Lemburg <mal@egenix.com>
parents: 15498
diff changeset
1039 mapped to themselves.
15867
f5a3d98a35bc Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 15706
diff changeset
1040
15706
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap
Marc-André Lemburg <mal@egenix.com>
parents: 15498
diff changeset
1041 """
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap
Marc-André Lemburg <mal@egenix.com>
parents: 15498
diff changeset
1042 res = {}
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap
Marc-André Lemburg <mal@egenix.com>
parents: 15498
diff changeset
1043 for i in rng:
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap
Marc-André Lemburg <mal@egenix.com>
parents: 15498
diff changeset
1044 res[i]=i
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap
Marc-André Lemburg <mal@egenix.com>
parents: 15498
diff changeset
1045 return res
bf9bb0e3dd34 This patch changes the default behaviour of the builtin charmap
Marc-André Lemburg <mal@egenix.com>
parents: 15498
diff changeset
1046
17843
3e256f9d99ad Moved the encoding map building logic from the individual mapping
Marc-André Lemburg <mal@egenix.com>
parents: 17839
diff changeset
1047 def make_encoding_map(decoding_map):
3e256f9d99ad Moved the encoding map building logic from the individual mapping
Marc-André Lemburg <mal@egenix.com>
parents: 17839
diff changeset
1048
3e256f9d99ad Moved the encoding map building logic from the individual mapping
Marc-André Lemburg <mal@egenix.com>
parents: 17839
diff changeset
1049 """ Creates an encoding map from a decoding map.
3e256f9d99ad Moved the encoding map building logic from the individual mapping
Marc-André Lemburg <mal@egenix.com>
parents: 17839
diff changeset
1050
27618
74d9604e9353 Fix typos.
Walter Dörwald <walter@livinglogic.de>
parents: 26906
diff changeset
1051 If a target mapping in the decoding map occurs multiple
17843
3e256f9d99ad Moved the encoding map building logic from the individual mapping
Marc-André Lemburg <mal@egenix.com>
parents: 17839
diff changeset
1052 times, then that target is mapped to None (undefined mapping),
3e256f9d99ad Moved the encoding map building logic from the individual mapping
Marc-André Lemburg <mal@egenix.com>
parents: 17839
diff changeset
1053 causing an exception when encountered by the charmap codec
3e256f9d99ad Moved the encoding map building logic from the individual mapping
Marc-André Lemburg <mal@egenix.com>
parents: 17839
diff changeset
1054 during translation.
3e256f9d99ad Moved the encoding map building logic from the individual mapping
Marc-André Lemburg <mal@egenix.com>
parents: 17839
diff changeset
1055
3e256f9d99ad Moved the encoding map building logic from the individual mapping
Marc-André Lemburg <mal@egenix.com>
parents: 17839
diff changeset
1056 One example where this happens is cp875.py which decodes
3e256f9d99ad Moved the encoding map building logic from the individual mapping
Marc-André Lemburg <mal@egenix.com>
parents: 17839
diff changeset
1057 multiple character to \u001a.
3e256f9d99ad Moved the encoding map building logic from the individual mapping
Marc-André Lemburg <mal@egenix.com>
parents: 17839
diff changeset
1058
3e256f9d99ad Moved the encoding map building logic from the individual mapping
Marc-André Lemburg <mal@egenix.com>
parents: 17839
diff changeset
1059 """
3e256f9d99ad Moved the encoding map building logic from the individual mapping
Marc-André Lemburg <mal@egenix.com>
parents: 17839
diff changeset
1060 m = {}
3e256f9d99ad Moved the encoding map building logic from the individual mapping
Marc-André Lemburg <mal@egenix.com>
parents: 17839
diff changeset
1061 for k,v in decoding_map.items():
23620
a1de0bab5475 SF 563203. Replaced 'has_key()' with 'in'.
Raymond Hettinger <python@rcn.com>
parents: 22149
diff changeset
1062 if not v in m:
17843
3e256f9d99ad Moved the encoding map building logic from the individual mapping
Marc-André Lemburg <mal@egenix.com>
parents: 17839
diff changeset
1063 m[v] = k
3e256f9d99ad Moved the encoding map building logic from the individual mapping
Marc-André Lemburg <mal@egenix.com>
parents: 17839
diff changeset
1064 else:
3e256f9d99ad Moved the encoding map building logic from the individual mapping
Marc-André Lemburg <mal@egenix.com>
parents: 17839
diff changeset
1065 m[v] = None
3e256f9d99ad Moved the encoding map building logic from the individual mapping
Marc-André Lemburg <mal@egenix.com>
parents: 17839
diff changeset
1066 return m
17978
5e87a08abb05 Whitespace normalization.
Tim Peters <tim.peters@gmail.com>
parents: 17843
diff changeset
1067
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 23658
diff changeset
1068 ### error handlers
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 23658
diff changeset
1069
34707
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607.
Martin v. Löwis <martin@v.loewis.de>
parents: 34387
diff changeset
1070 try:
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607.
Martin v. Löwis <martin@v.loewis.de>
parents: 34387
diff changeset
1071 strict_errors = lookup_error("strict")
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607.
Martin v. Löwis <martin@v.loewis.de>
parents: 34387
diff changeset
1072 ignore_errors = lookup_error("ignore")
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607.
Martin v. Löwis <martin@v.loewis.de>
parents: 34387
diff changeset
1073 replace_errors = lookup_error("replace")
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607.
Martin v. Löwis <martin@v.loewis.de>
parents: 34387
diff changeset
1074 xmlcharrefreplace_errors = lookup_error("xmlcharrefreplace")
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607.
Martin v. Löwis <martin@v.loewis.de>
parents: 34387
diff changeset
1075 backslashreplace_errors = lookup_error("backslashreplace")
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607.
Martin v. Löwis <martin@v.loewis.de>
parents: 34387
diff changeset
1076 except LookupError:
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607.
Martin v. Löwis <martin@v.loewis.de>
parents: 34387
diff changeset
1077 # In --disable-unicode builds, these error handler are missing
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607.
Martin v. Löwis <martin@v.loewis.de>
parents: 34387
diff changeset
1078 strict_errors = None
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607.
Martin v. Löwis <martin@v.loewis.de>
parents: 34387
diff changeset
1079 ignore_errors = None
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607.
Martin v. Löwis <martin@v.loewis.de>
parents: 34387
diff changeset
1080 replace_errors = None
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607.
Martin v. Löwis <martin@v.loewis.de>
parents: 34387
diff changeset
1081 xmlcharrefreplace_errors = None
0d44d9071e97 Build with --disable-unicode again. Fixes #1158607.
Martin v. Löwis <martin@v.loewis.de>
parents: 34387
diff changeset
1082 backslashreplace_errors = None
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 23658
diff changeset
1083
18904
5af1ee422575 Add dead imports of modules that are "magically" imported.
Martin v. Löwis <martin@v.loewis.de>
parents: 17978
diff changeset
1084 # Tell modulefinder that using codecs probably needs the encodings
5af1ee422575 Add dead imports of modules that are "magically" imported.
Martin v. Löwis <martin@v.loewis.de>
parents: 17978
diff changeset
1085 # package
5af1ee422575 Add dead imports of modules that are "magically" imported.
Martin v. Löwis <martin@v.loewis.de>
parents: 17978
diff changeset
1086 _false = 0
5af1ee422575 Add dead imports of modules that are "magically" imported.
Martin v. Löwis <martin@v.loewis.de>
parents: 17978
diff changeset
1087 if _false:
5af1ee422575 Add dead imports of modules that are "magically" imported.
Martin v. Löwis <martin@v.loewis.de>
parents: 17978
diff changeset
1088 import encodings
5af1ee422575 Add dead imports of modules that are "magically" imported.
Martin v. Löwis <martin@v.loewis.de>
parents: 17978
diff changeset
1089
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1090 ### Tests
12373
d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add
Guido van Rossum <guido@python.org>
parents: 12367
diff changeset
1091
11990
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1092 if __name__ == '__main__':
f104251b6f8f Module codecs -- Python Codec Registry, API and helpers. Written by
Guido van Rossum <guido@python.org>
parents:
diff changeset
1093
12367
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
1094 # Make stdout translate Latin-1 output into UTF-8 output
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
1095 sys.stdout = EncodedFile(sys.stdout, 'latin-1', 'utf-8')
12373
d63b57ac617d Deleted trailing whitespace. This is really a way to be able to add
Guido van Rossum <guido@python.org>
parents: 12367
diff changeset
1096
12367
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
1097 # Have stdin translate Latin-1 input into UTF-8 input
677e2147829a Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
1098 sys.stdin = EncodedFile(sys.stdin, 'utf-8', 'latin-1')