cpython-lru_cache-weakref: Doc/lib/libcodecs.tex annotate

annotate Doc/lib/libcodecs.tex @ 25135:0f0016632be4 legacy-trunk

_formatparam(), set_param(): RFC 2231 encoding support by Oleg Broytmann in SF patch #600096. Specifically, the former function now encodes the triplets, while the latter adds optional charset and language arguments.

author	Barry Warsaw <barry@python.org>
date	Fri, 06 Sep 2002 03:55:04 +0000
parents	0337dad8403e
children	5a17f0978326

rev	line source
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	1 \section{\module{codecs} ---
12274 559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	2 Codec registry and base classes}
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	3
12274 559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	4 \declaremodule{standard}{codecs}
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	5 \modulesynopsis{Encode and decode data and streams.}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	6 \moduleauthor{Marc-Andre Lemburg}{mal@lemburg.com}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	7 \sectionauthor{Marc-Andre Lemburg}{mal@lemburg.com}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	8
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	9
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	10 \index{Unicode}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	11 \index{Codecs}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	12 \indexii{Codecs}{encode}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	13 \indexii{Codecs}{decode}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	14 \index{streams}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	15 \indexii{stackable}{streams}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	16
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	17
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	18 This module defines base classes for standard Python codecs (encoders
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	19 and decoders) and provides access to the internal Python codec
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	20 registry which manages the codec and error handling lookup process.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	21
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	22 It defines the following functions:
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	23
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	24 \begin{funcdesc}{register}{search_function}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	25 Register a codec search function. Search functions are expected to
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	26 take one argument, the encoding name in all lower case letters, and
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	27 return a tuple of functions \code{(\var{encoder}, \var{decoder}, \var{stream_reader},
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	28 \var{stream_writer})} taking the following arguments:
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	29
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	30 \var{encoder} and \var{decoder}: These must be functions or methods
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	31 which have the same interface as the
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	32 \method{encode()}/\method{decode()} methods of Codec instances (see
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	33 Codec Interface). The functions/methods are expected to work in a
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	34 stateless mode.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	35
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	36 \var{stream_reader} and \var{stream_writer}: These have to be
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	37 factory functions providing the following interface:
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	38
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	39 \code{factory(\var{stream}, \var{errors}='strict')}
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	40
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	41 The factory functions must return objects providing the interfaces
12274 559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	42 defined by the base classes \class{StreamWriter} and
559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	43 \class{StreamReader}, respectively. Stream codecs can maintain
559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	44 state.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	45
12274 559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	46 Possible values for errors are \code{'strict'} (raise an exception
559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	47 in case of an encoding error), \code{'replace'} (replace malformed
559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	48 data with a suitable replacement marker, such as \character{?}) and
559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	49 \code{'ignore'} (ignore malformed data and continue without further
559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	50 notice).
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	51
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	52 In case a search function cannot find a given encoding, it should
12274 559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	53 return \code{None}.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	54 \end{funcdesc}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	55
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	56 \begin{funcdesc}{lookup}{encoding}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	57 Looks up a codec tuple in the Python codec registry and returns the
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	58 function tuple as defined above.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	59
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	60 Encodings are first looked up in the registry's cache. If not found,
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	61 the list of registered search functions is scanned. If no codecs tuple
12274 559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	62 is found, a \exception{LookupError} is raised. Otherwise, the codecs
559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	63 tuple is stored in the cache and returned to the caller.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	64 \end{funcdesc}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	65
23112 a3c982d412b4 typo Skip Montanaro <skip@pobox.com> parents: 23087 diff changeset	66 To simplify access to the various codecs, the module provides these
19982 bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	67 additional functions which use \function{lookup()} for the codec
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	68 lookup:
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	69
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	70 \begin{funcdesc}{getencoder}{encoding}
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	71 Lookup up the codec for the given encoding and return its encoder
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	72 function.
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	73
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	74 Raises a \exception{LookupError} in case the encoding cannot be found.
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	75 \end{funcdesc}
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	76
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	77 \begin{funcdesc}{getdecoder}{encoding}
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	78 Lookup up the codec for the given encoding and return its decoder
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	79 function.
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	80
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	81 Raises a \exception{LookupError} in case the encoding cannot be found.
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	82 \end{funcdesc}
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	83
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	84 \begin{funcdesc}{getreader}{encoding}
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	85 Lookup up the codec for the given encoding and return its StreamReader
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	86 class or factory function.
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	87
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	88 Raises a \exception{LookupError} in case the encoding cannot be found.
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	89 \end{funcdesc}
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	90
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	91 \begin{funcdesc}{getwriter}{encoding}
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	92 Lookup up the codec for the given encoding and return its StreamWriter
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	93 class or factory function.
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	94
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	95 Raises a \exception{LookupError} in case the encoding cannot be found.
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	96 \end{funcdesc}
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	97
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	98 To simplify working with encoded files or stream, the module
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	99 also defines these utility functions:
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	100
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	101 \begin{funcdesc}{register_error}{name, error_handler}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	102 Register the error handling function \var{error_handler} under the
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	103 name \var{name}. \vari{error_handler} will be called during encoding
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	104 and decoding in case of an error, when \var{name} is specified as the
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	105 errors parameter. \var{error_handler} will be called with an
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	106 \exception{UnicodeEncodeError}, \exception{UnicodeDecodeError} or
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	107 \exception{UnicodeTranslateError} instance and must return a tuple
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	108 with a replacement for the unencodable/undecodable part of the input
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	109 and a position where encoding/decoding should continue.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	110 \end{funcdesc}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	111
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	112 \begin{funcdesc}{lookup_error}{name}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	113 Return the error handler previously register under the name \var{name}.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	114
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	115 Raises a \exception{LookupError} in case the handler cannot be found.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	116 \end{funcdesc}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	117
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	118 \begin{funcdesc}{strict_errors}{exception}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	119 Implements the \code{strict} error handling.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	120 \end{funcdesc}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	121
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	122 \begin{funcdesc}{replace_errors}{exception}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	123 Implements the \code{replace} error handling.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	124 \end{funcdesc}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	125
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	126 \begin{funcdesc}{ignore_errors}{exception}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	127 Implements the \code{ignore} error handling.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	128 \end{funcdesc}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	129
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	130 \begin{funcdesc}{xmlcharrefreplace_errors_errors}{exception}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	131 Implements the \code{xmlcharrefreplace} error handling.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	132 \end{funcdesc}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	133
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	134 \begin{funcdesc}{backslashreplace_errors_errors}{exception}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	135 Implements the \code{backslashreplace} error handling.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	136 \end{funcdesc}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	137
13696 bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	138 \begin{funcdesc}{open}{filename, mode\optional{, encoding\optional{,
bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	139 errors\optional{, buffering}}}}
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	140 Open an encoded file using the given \var{mode} and return
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	141 a wrapped version providing transparent encoding/decoding.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	142
20580 376ee174ecd0 Use the \note and \warning macros where appropriate. Fred Drake <fdrake@acm.org> parents: 19982 diff changeset	143 \note{The wrapped version will only accept the object format
13696 bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	144 defined by the codecs, i.e.\ Unicode objects for most built-in
bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	145 codecs. Output is also codec-dependent and will usually be Unicode as
20580 376ee174ecd0 Use the \note and \warning macros where appropriate. Fred Drake <fdrake@acm.org> parents: 19982 diff changeset	146 well.}
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	147
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	148 \var{encoding} specifies the encoding which is to be used for the
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	149 the file.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	150
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	151 \var{errors} may be given to define the error handling. It defaults
13696 bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	152 to \code{'strict'} which causes a \exception{ValueError} to be raised
bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	153 in case an encoding error occurs.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	154
12274 559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	155 \var{buffering} has the same meaning as for the built-in
559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	156 \function{open()} function. It defaults to line buffered.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	157 \end{funcdesc}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	158
13696 bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	159 \begin{funcdesc}{EncodedFile}{file, input\optional{,
bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	160 output\optional{, errors}}}
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	161 Return a wrapped version of file which provides transparent
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	162 encoding translation.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	163
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	164 Strings written to the wrapped file are interpreted according to the
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	165 given \var{input} encoding and then written to the original file as
13696 bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	166 strings using the \var{output} encoding. The intermediate encoding will
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	167 usually be Unicode but depends on the specified codecs.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	168
13696 bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	169 If \var{output} is not given, it defaults to \var{input}.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	170
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	171 \var{errors} may be given to define the error handling. It defaults to
13696 bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	172 \code{'strict'}, which causes \exception{ValueError} to be raised in case
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	173 an encoding error occurs.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	174 \end{funcdesc}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	175
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	176 The module also provides the following constants which are useful
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	177 for reading and writing to platform dependent files:
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	178
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	179 \begin{datadesc}{BOM}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	180 \dataline{BOM_BE}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	181 \dataline{BOM_LE}
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	182 \dataline{BOM_UTF8}
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	183 \dataline{BOM_UTF16}
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	184 \dataline{BOM_UTF16_BE}
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	185 \dataline{BOM_UTF16_LE}
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	186 \dataline{BOM_UTF32}
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	187 \dataline{BOM_UTF32_BE}
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	188 \dataline{BOM_UTF32_LE}
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	189 These constants define various encodings of the Unicode byte order mark
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	190 (BOM) used in UTF-16 and UTF-32 data streams to indicate the byte order
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	191 used in the stream or file and in UTF-8 as a Unicode signature.
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	192 \constant{BOM_UTF16} is either \constant{BOM_UTF16_BE} or
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	193 \constant{BOM_UTF16_LE} depending on the platform's native byte order,
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	194 \constant{BOM} is an alias for \constant{BOM_UTF16}, \constant{BOM_LE}
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	195 for \constant{BOM_UTF16_LE} and \constant{BOM_BE} for \constant{BOM_UTF16_BE}.
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	196 The others represent the BOM in UTF-8 and UTF-32 encodings.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	197 \end{datadesc}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	198
16127 6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	199
6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	200 \begin{seealso}
6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	201 \seeurl{http://sourceforge.net/projects/python-codecs/}{A
6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	202 SourceForge project working on additional support for Asian
6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	203 codecs for use with Python. They are in the early stages of
6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	204 development at the time of this writing --- look in their
6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	205 FTP area for downloadable files.}
6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	206 \end{seealso}
6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	207
6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	208
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	209 \subsection{Codec Base Classes}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	210
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	211 The \module{codecs} defines a set of base classes which define the
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	212 interface and can also be used to easily write you own codecs for use
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	213 in Python.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	214
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	215 Each codec has to define four interfaces to make it usable as codec in
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	216 Python: stateless encoder, stateless decoder, stream reader and stream
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	217 writer. The stream reader and writers typically reuse the stateless
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	218 encoder/decoder to implement the file protocols.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	219
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	220 The \class{Codec} class defines the interface for stateless
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	221 encoders/decoders.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	222
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	223 To simplify and standardize error handling, the \method{encode()} and
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	224 \method{decode()} methods may implement different error handling
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	225 schemes by providing the \var{errors} string argument. The following
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	226 string values are defined and implemented by all standard Python
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	227 codecs:
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	228
16127 6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	229 \begin{tableii}{l\|l}{code}{Value}{Meaning}
6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	230 \lineii{'strict'}{Raise \exception{ValueError} (or a subclass);
6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	231 this is the default.}
6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	232 \lineii{'ignore'}{Ignore the character and continue with the next.}
6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	233 \lineii{'replace'}{Replace with a suitable replacement character;
6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	234 Python will use the official U+FFFD REPLACEMENT
6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	235 CHARACTER for the built-in Unicode codecs.}
6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	236 \end{tableii}
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	237
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	238
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	239 \subsubsection{Codec Objects \label{codec-objects}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	240
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	241 The \class{Codec} class defines these methods which also define the
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	242 function interfaces of the stateless encoder and decoder:
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	243
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	244 \begin{methoddesc}{encode}{input\optional{, errors}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	245 Encodes the object \var{input} and returns a tuple (output object,
23087 09db3f5d0bd0 added small clarification to the descriptions of encode() and decode() Skip Montanaro <skip@pobox.com> parents: 20580 diff changeset	246 length consumed). While codecs are not restricted to use with Unicode, in
09db3f5d0bd0 added small clarification to the descriptions of encode() and decode() Skip Montanaro <skip@pobox.com> parents: 20580 diff changeset	247 a Unicode context, encoding converts a Unicode object to a plain string
09db3f5d0bd0 added small clarification to the descriptions of encode() and decode() Skip Montanaro <skip@pobox.com> parents: 20580 diff changeset	248 using a particular character set encoding (e.g., \code{cp1252} or
09db3f5d0bd0 added small clarification to the descriptions of encode() and decode() Skip Montanaro <skip@pobox.com> parents: 20580 diff changeset	249 \code{iso-8859-1}).
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	250
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	251 \var{errors} defines the error handling to apply. It defaults to
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	252 \code{'strict'} handling.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	253
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	254 The method may not store state in the \class{Codec} instance. Use
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	255 \class{StreamCodec} for codecs which have to keep state in order to
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	256 make encoding/decoding efficient.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	257
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	258 The encoder must be able to handle zero length input and return an
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	259 empty object of the output object type in this situation.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	260 \end{methoddesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	261
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	262 \begin{methoddesc}{decode}{input\optional{, errors}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	263 Decodes the object \var{input} and returns a tuple (output object,
23087 09db3f5d0bd0 added small clarification to the descriptions of encode() and decode() Skip Montanaro <skip@pobox.com> parents: 20580 diff changeset	264 length consumed). In a Unicode context, decoding converts a plain string
09db3f5d0bd0 added small clarification to the descriptions of encode() and decode() Skip Montanaro <skip@pobox.com> parents: 20580 diff changeset	265 encoded using a particular character set encoding to a Unicode object.
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	266
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	267 \var{input} must be an object which provides the \code{bf_getreadbuf}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	268 buffer slot. Python strings, buffer objects and memory mapped files
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	269 are examples of objects providing this slot.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	270
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	271 \var{errors} defines the error handling to apply. It defaults to
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	272 \code{'strict'} handling.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	273
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	274 The method may not store state in the \class{Codec} instance. Use
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	275 \class{StreamCodec} for codecs which have to keep state in order to
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	276 make encoding/decoding efficient.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	277
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	278 The decoder must be able to handle zero length input and return an
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	279 empty object of the output object type in this situation.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	280 \end{methoddesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	281
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	282 The \class{StreamWriter} and \class{StreamReader} classes provide
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	283 generic working interfaces which can be used to implement new
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	284 encodings submodules very easily. See \module{encodings.utf_8} for an
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	285 example on how this is done.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	286
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	287
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	288 \subsubsection{StreamWriter Objects \label{stream-writer-objects}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	289
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	290 The \class{StreamWriter} class is a subclass of \class{Codec} and
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	291 defines the following methods which every stream writer must define in
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	292 order to be compatible to the Python codec registry.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	293
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	294 \begin{classdesc}{StreamWriter}{stream\optional{, errors}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	295 Constructor for a \class{StreamWriter} instance.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	296
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	297 All stream writers must provide this constructor interface. They are
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	298 free to add additional keyword arguments, but only the ones defined
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	299 here are used by the Python codec registry.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	300
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	301 \var{stream} must be a file-like object open for writing (binary)
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	302 data.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	303
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	304 The \class{StreamWriter} may implement different error handling
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	305 schemes by providing the \var{errors} keyword argument. These
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	306 parameters are defined:
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	307
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	308 \begin{itemize}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	309 \item \code{'strict'} Raise \exception{ValueError} (or a subclass);
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	310 this is the default.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	311 \item \code{'ignore'} Ignore the character and continue with the next.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	312 \item \code{'replace'} Replace with a suitable replacement character
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	313 \end{itemize}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	314 \end{classdesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	315
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	316 \begin{methoddesc}{write}{object}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	317 Writes the object's contents encoded to the stream.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	318 \end{methoddesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	319
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	320 \begin{methoddesc}{writelines}{list}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	321 Writes the concatenated list of strings to the stream (possibly by
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	322 reusing the \method{write()} method).
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	323 \end{methoddesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	324
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	325 \begin{methoddesc}{reset}{}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	326 Flushes and resets the codec buffers used for keeping state.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	327
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	328 Calling this method should ensure that the data on the output is put
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	329 into a clean state, that allows appending of new fresh data without
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	330 having to rescan the whole stream to recover state.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	331 \end{methoddesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	332
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	333 In addition to the above methods, the \class{StreamWriter} must also
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	334 inherit all other methods and attribute from the underlying stream.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	335
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	336
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	337 \subsubsection{StreamReader Objects \label{stream-reader-objects}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	338
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	339 The \class{StreamReader} class is a subclass of \class{Codec} and
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	340 defines the following methods which every stream reader must define in
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	341 order to be compatible to the Python codec registry.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	342
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	343 \begin{classdesc}{StreamReader}{stream\optional{, errors}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	344 Constructor for a \class{StreamReader} instance.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	345
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	346 All stream readers must provide this constructor interface. They are
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	347 free to add additional keyword arguments, but only the ones defined
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	348 here are used by the Python codec registry.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	349
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	350 \var{stream} must be a file-like object open for reading (binary)
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	351 data.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	352
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	353 The \class{StreamReader} may implement different error handling
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	354 schemes by providing the \var{errors} keyword argument. These
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	355 parameters are defined:
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	356
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	357 \begin{itemize}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	358 \item \code{'strict'} Raise \exception{ValueError} (or a subclass);
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	359 this is the default.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	360 \item \code{'ignore'} Ignore the character and continue with the next.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	361 \item \code{'replace'} Replace with a suitable replacement character.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	362 \end{itemize}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	363 \end{classdesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	364
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	365 \begin{methoddesc}{read}{\optional{size}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	366 Decodes data from the stream and returns the resulting object.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	367
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	368 \var{size} indicates the approximate maximum number of bytes to read
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	369 from the stream for decoding purposes. The decoder can modify this
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	370 setting as appropriate. The default value -1 indicates to read and
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	371 decode as much as possible. \var{size} is intended to prevent having
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	372 to decode huge files in one step.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	373
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	374 The method should use a greedy read strategy meaning that it should
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	375 read as much data as is allowed within the definition of the encoding
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	376 and the given size, e.g. if optional encoding endings or state
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	377 markers are available on the stream, these should be read too.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	378 \end{methoddesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	379
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	380 \begin{methoddesc}{readline}{[size]}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	381 Read one line from the input stream and return the
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	382 decoded data.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	383
20580 376ee174ecd0 Use the \note and \warning macros where appropriate. Fred Drake <fdrake@acm.org> parents: 19982 diff changeset	384 Unlike the \method{readlines()} method, this method inherits
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	385 the line breaking knowledge from the underlying stream's
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	386 \method{readline()} method -- there is currently no support for line
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	387 breaking using the codec decoder due to lack of line buffering.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	388 Sublcasses should however, if possible, try to implement this method
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	389 using their own knowledge of line breaking.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	390
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	391 \var{size}, if given, is passed as size argument to the stream's
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	392 \method{readline()} method.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	393 \end{methoddesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	394
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	395 \begin{methoddesc}{readlines}{[sizehint]}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	396 Read all lines available on the input stream and return them as list
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	397 of lines.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	398
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	399 Line breaks are implemented using the codec's decoder method and are
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	400 included in the list entries.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	401
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	402 \var{sizehint}, if given, is passed as \var{size} argument to the
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	403 stream's \method{read()} method.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	404 \end{methoddesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	405
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	406 \begin{methoddesc}{reset}{}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	407 Resets the codec buffers used for keeping state.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	408
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	409 Note that no stream repositioning should take place. This method is
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	410 primarily intended to be able to recover from decoding errors.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	411 \end{methoddesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	412
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	413 In addition to the above methods, the \class{StreamReader} must also
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	414 inherit all other methods and attribute from the underlying stream.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	415
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	416 The next two base classes are included for convenience. They are not
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	417 needed by the codec registry, but may provide useful in practice.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	418
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	419
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	420 \subsubsection{StreamReaderWriter Objects \label{stream-reader-writer}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	421
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	422 The \class{StreamReaderWriter} allows wrapping streams which work in
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	423 both read and write modes.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	424
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	425 The design is such that one can use the factory functions returned by
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	426 the \function{lookup()} function to construct the instance.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	427
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	428 \begin{classdesc}{StreamReaderWriter}{stream, Reader, Writer, errors}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	429 Creates a \class{StreamReaderWriter} instance.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	430 \var{stream} must be a file-like object.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	431 \var{Reader} and \var{Writer} must be factory functions or classes
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	432 providing the \class{StreamReader} and \class{StreamWriter} interface
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	433 resp.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	434 Error handling is done in the same way as defined for the
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	435 stream readers and writers.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	436 \end{classdesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	437
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	438 \class{StreamReaderWriter} instances define the combined interfaces of
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	439 \class{StreamReader} and \class{StreamWriter} classes. They inherit
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	440 all other methods and attribute from the underlying stream.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	441
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	442
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	443 \subsubsection{StreamRecoder Objects \label{stream-recoder-objects}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	444
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	445 The \class{StreamRecoder} provide a frontend - backend view of
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	446 encoding data which is sometimes useful when dealing with different
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	447 encoding environments.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	448
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	449 The design is such that one can use the factory functions returned by
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	450 the \function{lookup()} function to construct the instance.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	451
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	452 \begin{classdesc}{StreamRecoder}{stream, encode, decode,
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	453 Reader, Writer, errors}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	454 Creates a \class{StreamRecoder} instance which implements a two-way
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	455 conversion: \var{encode} and \var{decode} work on the frontend (the
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	456 input to \method{read()} and output of \method{write()}) while
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	457 \var{Reader} and \var{Writer} work on the backend (reading and
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	458 writing to the stream).
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	459
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	460 You can use these objects to do transparent direct recodings from
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	461 e.g.\ Latin-1 to UTF-8 and back.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	462
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	463 \var{stream} must be a file-like object.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	464
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	465 \var{encode}, \var{decode} must adhere to the \class{Codec}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	466 interface, \var{Reader}, \var{Writer} must be factory functions or
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	467 classes providing objects of the the \class{StreamReader} and
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	468 \class{StreamWriter} interface respectively.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	469
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	470 \var{encode} and \var{decode} are needed for the frontend
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	471 translation, \var{Reader} and \var{Writer} for the backend
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	472 translation. The intermediate format used is determined by the two
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	473 sets of codecs, e.g. the Unicode codecs will use Unicode as
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	474 intermediate encoding.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	475
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	476 Error handling is done in the same way as defined for the
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	477 stream readers and writers.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	478 \end{classdesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	479
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	480 \class{StreamRecoder} instances define the combined interfaces of
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	481 \class{StreamReader} and \class{StreamWriter} classes. They inherit
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	482 all other methods and attribute from the underlying stream.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	483

Mercurial > cpython-lru_cache-weakref

annotate Doc/lib/libcodecs.tex @ 25135:0f0016632be4 legacy-trunk