cpython-lru_cache-weakref: Doc/lib/libcodecs.tex annotate

annotate Doc/lib/libcodecs.tex @ 37850:ebe59cd18273

Use explicit relative import for an, ehm, relative import.

author	Thomas Wouters <thomas@python.org>
date	Tue, 18 Apr 2006 21:41:36 +0000
parents	0e36e7c66b8e
children	7fa4c29d7030

rev	line source
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	1 \section{\module{codecs} ---
12274 559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	2 Codec registry and base classes}
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	3
12274 559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	4 \declaremodule{standard}{codecs}
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	5 \modulesynopsis{Encode and decode data and streams.}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	6 \moduleauthor{Marc-Andre Lemburg}{mal@lemburg.com}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	7 \sectionauthor{Marc-Andre Lemburg}{mal@lemburg.com}
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	8 \sectionauthor{Martin v. L\"owis}{martin@v.loewis.de}
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	9
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	10 \index{Unicode}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	11 \index{Codecs}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	12 \indexii{Codecs}{encode}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	13 \indexii{Codecs}{decode}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	14 \index{streams}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	15 \indexii{stackable}{streams}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	16
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	17
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	18 This module defines base classes for standard Python codecs (encoders
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	19 and decoders) and provides access to the internal Python codec
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	20 registry which manages the codec and error handling lookup process.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	21
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	22 It defines the following functions:
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	23
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	24 \begin{funcdesc}{register}{search_function}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	25 Register a codec search function. Search functions are expected to
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	26 take one argument, the encoding name in all lower case letters, and
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	27 return a tuple of functions \code{(\var{encoder}, \var{decoder}, \var{stream_reader},
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	28 \var{stream_writer})} taking the following arguments:
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	29
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	30 \var{encoder} and \var{decoder}: These must be functions or methods
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	31 which have the same interface as the
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	32 \method{encode()}/\method{decode()} methods of Codec instances (see
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	33 Codec Interface). The functions/methods are expected to work in a
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	34 stateless mode.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	35
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	36 \var{stream_reader} and \var{stream_writer}: These have to be
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	37 factory functions providing the following interface:
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	38
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	39 \code{factory(\var{stream}, \var{errors}='strict')}
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	40
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	41 The factory functions must return objects providing the interfaces
12274 559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	42 defined by the base classes \class{StreamWriter} and
559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	43 \class{StreamReader}, respectively. Stream codecs can maintain
559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	44 state.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	45
12274 559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	46 Possible values for errors are \code{'strict'} (raise an exception
559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	47 in case of an encoding error), \code{'replace'} (replace malformed
26294 6d0cd80e2804 Document additional error handling names available through PEP 293. Walter Dörwald <walter@livinglogic.de> parents: 26099 diff changeset	48 data with a suitable replacement marker, such as \character{?}),
12274 559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	49 \code{'ignore'} (ignore malformed data and continue without further
26294 6d0cd80e2804 Document additional error handling names available through PEP 293. Walter Dörwald <walter@livinglogic.de> parents: 26099 diff changeset	50 notice), \code{'xmlcharrefreplace'} (replace with the appropriate XML
6d0cd80e2804 Document additional error handling names available through PEP 293. Walter Dörwald <walter@livinglogic.de> parents: 26099 diff changeset	51 character reference (for encoding only)) and \code{'backslashreplace'}
6d0cd80e2804 Document additional error handling names available through PEP 293. Walter Dörwald <walter@livinglogic.de> parents: 26099 diff changeset	52 (replace with backslashed escape sequences (for encoding only)) as
6d0cd80e2804 Document additional error handling names available through PEP 293. Walter Dörwald <walter@livinglogic.de> parents: 26099 diff changeset	53 well as any other error handling name defined via
6d0cd80e2804 Document additional error handling names available through PEP 293. Walter Dörwald <walter@livinglogic.de> parents: 26099 diff changeset	54 \function{register_error()}.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	55
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	56 In case a search function cannot find a given encoding, it should
12274 559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	57 return \code{None}.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	58 \end{funcdesc}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	59
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	60 \begin{funcdesc}{lookup}{encoding}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	61 Looks up a codec tuple in the Python codec registry and returns the
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	62 function tuple as defined above.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	63
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	64 Encodings are first looked up in the registry's cache. If not found,
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	65 the list of registered search functions is scanned. If no codecs tuple
12274 559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	66 is found, a \exception{LookupError} is raised. Otherwise, the codecs
559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	67 tuple is stored in the cache and returned to the caller.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	68 \end{funcdesc}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	69
23112 a3c982d412b4 typo Skip Montanaro <skip@pobox.com> parents: 23087 diff changeset	70 To simplify access to the various codecs, the module provides these
19982 bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	71 additional functions which use \function{lookup()} for the codec
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	72 lookup:
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	73
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	74 \begin{funcdesc}{getencoder}{encoding}
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	75 Lookup up the codec for the given encoding and return its encoder
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	76 function.
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	77
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	78 Raises a \exception{LookupError} in case the encoding cannot be found.
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	79 \end{funcdesc}
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	80
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	81 \begin{funcdesc}{getdecoder}{encoding}
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	82 Lookup up the codec for the given encoding and return its decoder
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	83 function.
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	84
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	85 Raises a \exception{LookupError} in case the encoding cannot be found.
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	86 \end{funcdesc}
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	87
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	88 \begin{funcdesc}{getreader}{encoding}
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	89 Lookup up the codec for the given encoding and return its StreamReader
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	90 class or factory function.
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	91
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	92 Raises a \exception{LookupError} in case the encoding cannot be found.
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	93 \end{funcdesc}
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	94
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	95 \begin{funcdesc}{getwriter}{encoding}
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	96 Lookup up the codec for the given encoding and return its StreamWriter
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	97 class or factory function.
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	98
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	99 Raises a \exception{LookupError} in case the encoding cannot be found.
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	100 \end{funcdesc}
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	101
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	102 \begin{funcdesc}{register_error}{name, error_handler}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	103 Register the error handling function \var{error_handler} under the
25172 5a17f0978326 Fix typo. Close SF Bug 606354. Raymond Hettinger <python@rcn.com> parents: 25091 diff changeset	104 name \var{name}. \var{error_handler} will be called during encoding
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	105 and decoding in case of an error, when \var{name} is specified as the
27555 c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	106 errors parameter.
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	107
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	108 For encoding \var{error_handler} will be called with a
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	109 \exception{UnicodeEncodeError} instance, which contains information about
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	110 the location of the error. The error handler must either raise this or
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	111 a different exception or return a tuple with a replacement for the
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	112 unencodable part of the input and a position where encoding should
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	113 continue. The encoder will encode the replacement and continue encoding
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	114 the original input at the specified position. Negative position values
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	115 will be treated as being relative to the end of the input string. If the
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	116 resulting position is out of bound an IndexError will be raised.
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	117
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	118 Decoding and translating works similar, except \exception{UnicodeDecodeError}
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	119 or \exception{UnicodeTranslateError} will be passed to the handler and
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	120 that the replacement from the error handler will be put into the output
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	121 directly.
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	122 \end{funcdesc}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	123
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	124 \begin{funcdesc}{lookup_error}{name}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	125 Return the error handler previously register under the name \var{name}.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	126
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	127 Raises a \exception{LookupError} in case the handler cannot be found.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	128 \end{funcdesc}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	129
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	130 \begin{funcdesc}{strict_errors}{exception}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	131 Implements the \code{strict} error handling.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	132 \end{funcdesc}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	133
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	134 \begin{funcdesc}{replace_errors}{exception}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	135 Implements the \code{replace} error handling.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	136 \end{funcdesc}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	137
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	138 \begin{funcdesc}{ignore_errors}{exception}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	139 Implements the \code{ignore} error handling.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	140 \end{funcdesc}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	141
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	142 \begin{funcdesc}{xmlcharrefreplace_errors_errors}{exception}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	143 Implements the \code{xmlcharrefreplace} error handling.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	144 \end{funcdesc}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	145
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	146 \begin{funcdesc}{backslashreplace_errors_errors}{exception}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	147 Implements the \code{backslashreplace} error handling.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	148 \end{funcdesc}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	149
25989 611af79c0e89 Move introductory sentence to where it belongs. Walter Dörwald <walter@livinglogic.de> parents: 25172 diff changeset	150 To simplify working with encoded files or stream, the module
611af79c0e89 Move introductory sentence to where it belongs. Walter Dörwald <walter@livinglogic.de> parents: 25172 diff changeset	151 also defines these utility functions:
611af79c0e89 Move introductory sentence to where it belongs. Walter Dörwald <walter@livinglogic.de> parents: 25172 diff changeset	152
13696 bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	153 \begin{funcdesc}{open}{filename, mode\optional{, encoding\optional{,
bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	154 errors\optional{, buffering}}}}
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	155 Open an encoded file using the given \var{mode} and return
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	156 a wrapped version providing transparent encoding/decoding.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	157
20580 376ee174ecd0 Use the \note and \warning macros where appropriate. Fred Drake <fdrake@acm.org> parents: 19982 diff changeset	158 \note{The wrapped version will only accept the object format
13696 bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	159 defined by the codecs, i.e.\ Unicode objects for most built-in
bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	160 codecs. Output is also codec-dependent and will usually be Unicode as
20580 376ee174ecd0 Use the \note and \warning macros where appropriate. Fred Drake <fdrake@acm.org> parents: 19982 diff changeset	161 well.}
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	162
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	163 \var{encoding} specifies the encoding which is to be used for the
30505 f4a01c405a44 Fix doubled word errors. Raymond Hettinger <python@rcn.com> parents: 30273 diff changeset	164 file.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	165
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	166 \var{errors} may be given to define the error handling. It defaults
13696 bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	167 to \code{'strict'} which causes a \exception{ValueError} to be raised
bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	168 in case an encoding error occurs.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	169
12274 559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	170 \var{buffering} has the same meaning as for the built-in
559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	171 \function{open()} function. It defaults to line buffered.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	172 \end{funcdesc}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	173
13696 bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	174 \begin{funcdesc}{EncodedFile}{file, input\optional{,
bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	175 output\optional{, errors}}}
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	176 Return a wrapped version of file which provides transparent
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	177 encoding translation.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	178
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	179 Strings written to the wrapped file are interpreted according to the
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	180 given \var{input} encoding and then written to the original file as
13696 bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	181 strings using the \var{output} encoding. The intermediate encoding will
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	182 usually be Unicode but depends on the specified codecs.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	183
13696 bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	184 If \var{output} is not given, it defaults to \var{input}.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	185
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	186 \var{errors} may be given to define the error handling. It defaults to
13696 bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	187 \code{'strict'}, which causes \exception{ValueError} to be raised in case
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	188 an encoding error occurs.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	189 \end{funcdesc}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	190
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	191 The module also provides the following constants which are useful
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	192 for reading and writing to platform dependent files:
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	193
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	194 \begin{datadesc}{BOM}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	195 \dataline{BOM_BE}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	196 \dataline{BOM_LE}
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	197 \dataline{BOM_UTF8}
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	198 \dataline{BOM_UTF16}
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	199 \dataline{BOM_UTF16_BE}
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	200 \dataline{BOM_UTF16_LE}
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	201 \dataline{BOM_UTF32}
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	202 \dataline{BOM_UTF32_BE}
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	203 \dataline{BOM_UTF32_LE}
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	204 These constants define various encodings of the Unicode byte order mark
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	205 (BOM) used in UTF-16 and UTF-32 data streams to indicate the byte order
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	206 used in the stream or file and in UTF-8 as a Unicode signature.
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	207 \constant{BOM_UTF16} is either \constant{BOM_UTF16_BE} or
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	208 \constant{BOM_UTF16_LE} depending on the platform's native byte order,
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	209 \constant{BOM} is an alias for \constant{BOM_UTF16}, \constant{BOM_LE}
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	210 for \constant{BOM_UTF16_LE} and \constant{BOM_BE} for \constant{BOM_UTF16_BE}.
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	211 The others represent the BOM in UTF-8 and UTF-32 encodings.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	212 \end{datadesc}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	213
16127 6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	214
35951 90797233208e SF patch #1364946: Add a reference link from the dcoumentation of the encode Walter Dörwald <walter@livinglogic.de> parents: 35739 diff changeset	215 \subsection{Codec Base Classes \label{codec-base-classes}}
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	216
35739 d0399379d0cb add missing word Fred Drake <fdrake@acm.org> parents: 35710 diff changeset	217 The \module{codecs} module defines a set of base classes which define the
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	218 interface and can also be used to easily write you own codecs for use
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	219 in Python.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	220
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	221 Each codec has to define four interfaces to make it usable as codec in
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	222 Python: stateless encoder, stateless decoder, stream reader and stream
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	223 writer. The stream reader and writers typically reuse the stateless
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	224 encoder/decoder to implement the file protocols.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	225
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	226 The \class{Codec} class defines the interface for stateless
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	227 encoders/decoders.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	228
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	229 To simplify and standardize error handling, the \method{encode()} and
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	230 \method{decode()} methods may implement different error handling
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	231 schemes by providing the \var{errors} string argument. The following
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	232 string values are defined and implemented by all standard Python
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	233 codecs:
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	234
16127 6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	235 \begin{tableii}{l\|l}{code}{Value}{Meaning}
26099 135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	236 \lineii{'strict'}{Raise \exception{UnicodeError} (or a subclass);
16127 6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	237 this is the default.}
6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	238 \lineii{'ignore'}{Ignore the character and continue with the next.}
6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	239 \lineii{'replace'}{Replace with a suitable replacement character;
6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	240 Python will use the official U+FFFD REPLACEMENT
26099 135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	241 CHARACTER for the built-in Unicode codecs on
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	242 decoding and '?' on encoding.}
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	243 \lineii{'xmlcharrefreplace'}{Replace with the appropriate XML
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	244 character reference (only for encoding).}
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	245 \lineii{'backslashreplace'}{Replace with backslashed escape sequences
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	246 (only for encoding).}
16127 6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	247 \end{tableii}
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	248
26099 135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	249 The set of allowed values can be extended via \method{register_error}.
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	250
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	251
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	252 \subsubsection{Codec Objects \label{codec-objects}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	253
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	254 The \class{Codec} class defines these methods which also define the
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	255 function interfaces of the stateless encoder and decoder:
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	256
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	257 \begin{methoddesc}{encode}{input\optional{, errors}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	258 Encodes the object \var{input} and returns a tuple (output object,
23087 09db3f5d0bd0 added small clarification to the descriptions of encode() and decode() Skip Montanaro <skip@pobox.com> parents: 20580 diff changeset	259 length consumed). While codecs are not restricted to use with Unicode, in
09db3f5d0bd0 added small clarification to the descriptions of encode() and decode() Skip Montanaro <skip@pobox.com> parents: 20580 diff changeset	260 a Unicode context, encoding converts a Unicode object to a plain string
09db3f5d0bd0 added small clarification to the descriptions of encode() and decode() Skip Montanaro <skip@pobox.com> parents: 20580 diff changeset	261 using a particular character set encoding (e.g., \code{cp1252} or
09db3f5d0bd0 added small clarification to the descriptions of encode() and decode() Skip Montanaro <skip@pobox.com> parents: 20580 diff changeset	262 \code{iso-8859-1}).
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	263
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	264 \var{errors} defines the error handling to apply. It defaults to
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	265 \code{'strict'} handling.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	266
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	267 The method may not store state in the \class{Codec} instance. Use
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	268 \class{StreamCodec} for codecs which have to keep state in order to
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	269 make encoding/decoding efficient.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	270
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	271 The encoder must be able to handle zero length input and return an
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	272 empty object of the output object type in this situation.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	273 \end{methoddesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	274
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	275 \begin{methoddesc}{decode}{input\optional{, errors}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	276 Decodes the object \var{input} and returns a tuple (output object,
23087 09db3f5d0bd0 added small clarification to the descriptions of encode() and decode() Skip Montanaro <skip@pobox.com> parents: 20580 diff changeset	277 length consumed). In a Unicode context, decoding converts a plain string
09db3f5d0bd0 added small clarification to the descriptions of encode() and decode() Skip Montanaro <skip@pobox.com> parents: 20580 diff changeset	278 encoded using a particular character set encoding to a Unicode object.
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	279
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	280 \var{input} must be an object which provides the \code{bf_getreadbuf}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	281 buffer slot. Python strings, buffer objects and memory mapped files
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	282 are examples of objects providing this slot.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	283
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	284 \var{errors} defines the error handling to apply. It defaults to
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	285 \code{'strict'} handling.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	286
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	287 The method may not store state in the \class{Codec} instance. Use
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	288 \class{StreamCodec} for codecs which have to keep state in order to
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	289 make encoding/decoding efficient.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	290
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	291 The decoder must be able to handle zero length input and return an
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	292 empty object of the output object type in this situation.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	293 \end{methoddesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	294
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	295 The \class{StreamWriter} and \class{StreamReader} classes provide
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	296 generic working interfaces which can be used to implement new
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	297 encodings submodules very easily. See \module{encodings.utf_8} for an
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	298 example on how this is done.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	299
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	300
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	301 \subsubsection{StreamWriter Objects \label{stream-writer-objects}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	302
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	303 The \class{StreamWriter} class is a subclass of \class{Codec} and
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	304 defines the following methods which every stream writer must define in
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	305 order to be compatible to the Python codec registry.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	306
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	307 \begin{classdesc}{StreamWriter}{stream\optional{, errors}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	308 Constructor for a \class{StreamWriter} instance.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	309
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	310 All stream writers must provide this constructor interface. They are
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	311 free to add additional keyword arguments, but only the ones defined
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	312 here are used by the Python codec registry.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	313
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	314 \var{stream} must be a file-like object open for writing (binary)
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	315 data.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	316
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	317 The \class{StreamWriter} may implement different error handling
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	318 schemes by providing the \var{errors} keyword argument. These
26099 135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	319 parameters are predefined:
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	320
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	321 \begin{itemize}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	322 \item \code{'strict'} Raise \exception{ValueError} (or a subclass);
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	323 this is the default.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	324 \item \code{'ignore'} Ignore the character and continue with the next.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	325 \item \code{'replace'} Replace with a suitable replacement character
26099 135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	326 \item \code{'xmlcharrefreplace'} Replace with the appropriate XML
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	327 character reference
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	328 \item \code{'backslashreplace'} Replace with backslashed escape sequences.
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	329 \end{itemize}
26099 135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	330
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	331 The \var{errors} argument will be assigned to an attribute of the
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	332 same name. Assigning to this attribute makes it possible to switch
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	333 between different error handling strategies during the lifetime
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	334 of the \class{StreamWriter} object.
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	335
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	336 The set of allowed values for the \var{errors} argument can
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	337 be extended with \function{register_error()}.
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	338 \end{classdesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	339
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	340 \begin{methoddesc}{write}{object}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	341 Writes the object's contents encoded to the stream.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	342 \end{methoddesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	343
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	344 \begin{methoddesc}{writelines}{list}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	345 Writes the concatenated list of strings to the stream (possibly by
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	346 reusing the \method{write()} method).
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	347 \end{methoddesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	348
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	349 \begin{methoddesc}{reset}{}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	350 Flushes and resets the codec buffers used for keeping state.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	351
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	352 Calling this method should ensure that the data on the output is put
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	353 into a clean state, that allows appending of new fresh data without
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	354 having to rescan the whole stream to recover state.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	355 \end{methoddesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	356
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	357 In addition to the above methods, the \class{StreamWriter} must also
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	358 inherit all other methods and attribute from the underlying stream.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	359
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	360
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	361 \subsubsection{StreamReader Objects \label{stream-reader-objects}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	362
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	363 The \class{StreamReader} class is a subclass of \class{Codec} and
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	364 defines the following methods which every stream reader must define in
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	365 order to be compatible to the Python codec registry.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	366
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	367 \begin{classdesc}{StreamReader}{stream\optional{, errors}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	368 Constructor for a \class{StreamReader} instance.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	369
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	370 All stream readers must provide this constructor interface. They are
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	371 free to add additional keyword arguments, but only the ones defined
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	372 here are used by the Python codec registry.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	373
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	374 \var{stream} must be a file-like object open for reading (binary)
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	375 data.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	376
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	377 The \class{StreamReader} may implement different error handling
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	378 schemes by providing the \var{errors} keyword argument. These
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	379 parameters are defined:
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	380
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	381 \begin{itemize}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	382 \item \code{'strict'} Raise \exception{ValueError} (or a subclass);
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	383 this is the default.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	384 \item \code{'ignore'} Ignore the character and continue with the next.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	385 \item \code{'replace'} Replace with a suitable replacement character.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	386 \end{itemize}
26099 135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	387
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	388 The \var{errors} argument will be assigned to an attribute of the
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	389 same name. Assigning to this attribute makes it possible to switch
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	390 between different error handling strategies during the lifetime
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	391 of the \class{StreamReader} object.
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	392
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	393 The set of allowed values for the \var{errors} argument can
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	394 be extended with \function{register_error()}.
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	395 \end{classdesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	396
35416 4896f8334bff Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 34296 diff changeset	397 \begin{methoddesc}{read}{\optional{size\optional{, chars, \optional{firstline}}}}
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	398 Decodes data from the stream and returns the resulting object.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	399
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32727 diff changeset	400 \var{chars} indicates the number of characters to read from the
33437 fac2a53906f1 fix typo in markup Fred Drake <fdrake@acm.org> parents: 33428 diff changeset	401 stream. \function{read()} will never return more than \var{chars}
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32727 diff changeset	402 characters, but it might return less, if there are not enough
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32727 diff changeset	403 characters available.
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32727 diff changeset	404
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	405 \var{size} indicates the approximate maximum number of bytes to read
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	406 from the stream for decoding purposes. The decoder can modify this
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	407 setting as appropriate. The default value -1 indicates to read and
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	408 decode as much as possible. \var{size} is intended to prevent having
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	409 to decode huge files in one step.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	410
35416 4896f8334bff Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 34296 diff changeset	411 \var{firstline} indicates that it would be sufficient to only return
4896f8334bff Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 34296 diff changeset	412 the first line, if there are decoding errors on later lines.
4896f8334bff Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 34296 diff changeset	413
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	414 The method should use a greedy read strategy meaning that it should
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	415 read as much data as is allowed within the definition of the encoding
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	416 and the given size, e.g. if optional encoding endings or state
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	417 markers are available on the stream, these should be read too.
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32727 diff changeset	418
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32727 diff changeset	419 \versionchanged[\var{chars} argument added]{2.4}
35416 4896f8334bff Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 34296 diff changeset	420 \versionchanged[\var{firstline} argument added]{2.4.2}
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	421 \end{methoddesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	422
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32727 diff changeset	423 \begin{methoddesc}{readline}{\optional{size\optional{, keepends}}}
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	424 Read one line from the input stream and return the
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	425 decoded data.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	426
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	427 \var{size}, if given, is passed as size argument to the stream's
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	428 \method{readline()} method.
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32727 diff changeset	429
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32727 diff changeset	430 If \var{keepends} is false lineends will be stripped from the
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32727 diff changeset	431 lines returned.
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32727 diff changeset	432
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32727 diff changeset	433 \versionchanged[\var{keepends} argument added]{2.4}
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	434 \end{methoddesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	435
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32727 diff changeset	436 \begin{methoddesc}{readlines}{\optional{sizehint\optional{, keepends}}}
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	437 Read all lines available on the input stream and return them as list
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	438 of lines.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	439
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	440 Line breaks are implemented using the codec's decoder method and are
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32727 diff changeset	441 included in the list entries if \var{keepends} is true.
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	442
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	443 \var{sizehint}, if given, is passed as \var{size} argument to the
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	444 stream's \method{read()} method.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	445 \end{methoddesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	446
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	447 \begin{methoddesc}{reset}{}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	448 Resets the codec buffers used for keeping state.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	449
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	450 Note that no stream repositioning should take place. This method is
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	451 primarily intended to be able to recover from decoding errors.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	452 \end{methoddesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	453
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	454 In addition to the above methods, the \class{StreamReader} must also
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	455 inherit all other methods and attribute from the underlying stream.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	456
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	457 The next two base classes are included for convenience. They are not
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	458 needed by the codec registry, but may provide useful in practice.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	459
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	460
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	461 \subsubsection{StreamReaderWriter Objects \label{stream-reader-writer}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	462
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	463 The \class{StreamReaderWriter} allows wrapping streams which work in
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	464 both read and write modes.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	465
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	466 The design is such that one can use the factory functions returned by
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	467 the \function{lookup()} function to construct the instance.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	468
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	469 \begin{classdesc}{StreamReaderWriter}{stream, Reader, Writer, errors}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	470 Creates a \class{StreamReaderWriter} instance.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	471 \var{stream} must be a file-like object.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	472 \var{Reader} and \var{Writer} must be factory functions or classes
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	473 providing the \class{StreamReader} and \class{StreamWriter} interface
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	474 resp.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	475 Error handling is done in the same way as defined for the
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	476 stream readers and writers.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	477 \end{classdesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	478
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	479 \class{StreamReaderWriter} instances define the combined interfaces of
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	480 \class{StreamReader} and \class{StreamWriter} classes. They inherit
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	481 all other methods and attribute from the underlying stream.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	482
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	483
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	484 \subsubsection{StreamRecoder Objects \label{stream-recoder-objects}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	485
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	486 The \class{StreamRecoder} provide a frontend - backend view of
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	487 encoding data which is sometimes useful when dealing with different
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	488 encoding environments.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	489
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	490 The design is such that one can use the factory functions returned by
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	491 the \function{lookup()} function to construct the instance.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	492
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	493 \begin{classdesc}{StreamRecoder}{stream, encode, decode,
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	494 Reader, Writer, errors}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	495 Creates a \class{StreamRecoder} instance which implements a two-way
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	496 conversion: \var{encode} and \var{decode} work on the frontend (the
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	497 input to \method{read()} and output of \method{write()}) while
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	498 \var{Reader} and \var{Writer} work on the backend (reading and
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	499 writing to the stream).
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	500
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	501 You can use these objects to do transparent direct recodings from
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	502 e.g.\ Latin-1 to UTF-8 and back.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	503
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	504 \var{stream} must be a file-like object.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	505
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	506 \var{encode}, \var{decode} must adhere to the \class{Codec}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	507 interface, \var{Reader}, \var{Writer} must be factory functions or
30134 523c3d61dbdd SF patch#786531 'the the' typo. Contributed by George Yoshida Raymond Hettinger <python@rcn.com> parents: 29894 diff changeset	508 classes providing objects of the \class{StreamReader} and
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	509 \class{StreamWriter} interface respectively.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	510
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	511 \var{encode} and \var{decode} are needed for the frontend
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	512 translation, \var{Reader} and \var{Writer} for the backend
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	513 translation. The intermediate format used is determined by the two
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	514 sets of codecs, e.g. the Unicode codecs will use Unicode as
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	515 intermediate encoding.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	516
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	517 Error handling is done in the same way as defined for the
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	518 stream readers and writers.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	519 \end{classdesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	520
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	521 \class{StreamRecoder} instances define the combined interfaces of
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	522 \class{StreamReader} and \class{StreamWriter} classes. They inherit
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	523 all other methods and attribute from the underlying stream.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	524
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	525 \subsection{Encodings and Unicode\label{encodings-overview}}
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	526
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	527 Unicode strings are stored internally as sequences of codepoints (to
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	528 be precise as \ctype{Py_UNICODE} arrays). Depending on the way Python is
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	529 compiled (either via \longprogramopt{enable-unicode=ucs2} or
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	530 \longprogramopt{enable-unicode=ucs4}, with the former being the default)
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	531 \ctype{Py_UNICODE} is either a 16-bit or
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	532 32-bit data type. Once a Unicode object is used outside of CPU and
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	533 memory, CPU endianness and how these arrays are stored as bytes become
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	534 an issue. Transforming a unicode object into a sequence of bytes is
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	535 called encoding and recreating the unicode object from the sequence of
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	536 bytes is known as decoding. There are many different methods how this
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	537 transformation can be done (these methods are also called encodings).
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	538 The simplest method is to map the codepoints 0-255 to the bytes
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	539 \code{0x0}-\code{0xff}. This means that a unicode object that contains
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	540 codepoints above \code{U+00FF} can't be encoded with this method (which
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	541 is called \code{'latin-1'} or \code{'iso-8859-1'}). unicode.encode() will
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	542 raise a UnicodeEncodeError that looks like this: \samp{UnicodeEncodeError:
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	543 'latin-1' codec can't encode character u'\e u1234' in position 3: ordinal
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	544 not in range(256)}.
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	545
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	546 There's another group of encodings (the so called charmap encodings)
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	547 that choose a different subset of all unicode code points and how
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	548 these codepoints are mapped to the bytes \code{0x0}-\code{0xff.}
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	549 To see how this is done simply open e.g. \file{encodings/cp1252.py}
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	550 (which is an encoding that is used primarily on Windows).
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	551 There's a string constant with 256 characters that shows you which
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	552 character is mapped to which byte value.
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	553
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	554 All of these encodings can only encode 256 of the 65536 (or 1114111)
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	555 codepoints defined in unicode. A simple and straightforward way that
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	556 can store each Unicode code point, is to store each codepoint as two
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	557 consecutive bytes. There are two possibilities: Store the bytes in big
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	558 endian or in little endian order. These two encodings are called
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	559 UTF-16-BE and UTF-16-LE respectively. Their disadvantage is that if
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	560 e.g. you use UTF-16-BE on a little endian machine you will always have
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	561 to swap bytes on encoding and decoding. UTF-16 avoids this problem:
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	562 Bytes will always be in natural endianness. When these bytes are read
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	563 by a CPU with a different endianness, then bytes have to be swapped
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	564 though. To be able to detect the endianness of a UTF-16 byte sequence,
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	565 there's the so called BOM (the "Byte Order Mark"). This is the Unicode
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	566 character \code{U+FEFF}. This character will be prepended to every UTF-16
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	567 byte sequence. The byte swapped version of this character (\code{0xFFFE}) is
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	568 an illegal character that may not appear in a Unicode text. So when
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	569 the first character in an UTF-16 byte sequence appears to be a \code{U+FFFE}
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	570 the bytes have to be swapped on decoding. Unfortunately upto Unicode
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	571 4.0 the character \code{U+FEFF} had a second purpose as a \samp{ZERO WIDTH
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	572 NO-BREAK SPACE}: A character that has no width and doesn't allow a
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	573 word to be split. It can e.g. be used to give hints to a ligature
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	574 algorithm. With Unicode 4.0 using \code{U+FEFF} as a \samp{ZERO WIDTH NO-BREAK
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	575 SPACE} has been deprecated (with \code{U+2060} (\samp{WORD JOINER}) assuming
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	576 this role). Nevertheless Unicode software still must be able to handle
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	577 \code{U+FEFF} in both roles: As a BOM it's a device to determine the storage
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	578 layout of the encoded bytes, and vanishes once the byte sequence has
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	579 been decoded into a Unicode string; as a \samp{ZERO WIDTH NO-BREAK SPACE}
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	580 it's a normal character that will be decoded like any other.
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	581
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	582 There's another encoding that is able to encoding the full range of
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	583 Unicode characters: UTF-8. UTF-8 is an 8bit encoding, which means
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	584 there are no issues with byte order in UTF-8. Each byte in a UTF-8
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	585 byte sequence consists of two parts: Marker bits (the most significant
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	586 bits) and payload bits. The marker bits are a sequence of zero to six
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	587 1 bits followed by a 0 bit. Unicode characters are encoded like this
36256 6cf7d04dcc1f Fix typos. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	588 (with x being payload bits, which when concatenated give the Unicode
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	589 character):
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	590
36637 d0ebcb4f6031 Fix table header. Walter Dörwald <walter@livinglogic.de> parents: 36389 diff changeset	591 \begin{tableii}{l\|l}{textrm}{Range}{Encoding}
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	592 \lineii{\code{U-00000000} ... \code{U-0000007F}}{0xxxxxxx}
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	593 \lineii{\code{U-00000080} ... \code{U-000007FF}}{110xxxxx 10xxxxxx}
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	594 \lineii{\code{U-00000800} ... \code{U-0000FFFF}}{1110xxxx 10xxxxxx 10xxxxxx}
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	595 \lineii{\code{U-00010000} ... \code{U-001FFFFF}}{11110xxx 10xxxxxx 10xxxxxx 10xxxxxx}
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	596 \lineii{\code{U-00200000} ... \code{U-03FFFFFF}}{111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx}
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	597 \lineii{\code{U-04000000} ... \code{U-7FFFFFFF}}{1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx}
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	598 \end{tableii}
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	599
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	600 The least significant bit of the Unicode character is the rightmost x
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	601 bit.
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	602
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	603 As UTF-8 is an 8bit encoding no BOM is required and any \code{U+FEFF}
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	604 character in the decoded Unicode string (even if it's the first
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	605 character) is treated as a \samp{ZERO WIDTH NO-BREAK SPACE}.
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	606
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	607 Without external information it's impossible to reliably determine
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	608 which encoding was used for encoding a Unicode string. Each charmap
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	609 encoding can decode any random byte sequence. However that's not
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	610 possible with UTF-8, as UTF-8 byte sequences have a structure that
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	611 doesn't allow arbitrary byte sequence. To increase the reliability
36256 6cf7d04dcc1f Fix typos. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	612 with which a UTF-8 encoding can be detected, Microsoft invented a
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	613 variant of UTF-8 (that Python 2.5 calls \code{"utf-8-sig"}) for its Notepad
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	614 program: Before any of the Unicode characters is written to the file,
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	615 a UTF-8 encoded BOM (which looks like this as a byte sequence: \code{0xef},
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	616 \code{0xbb}, \code{0xbf}) is written. As it's rather improbably that any
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	617 charmap encoded file starts with these byte values (which would e.g. map to
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	618
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	619 LATIN SMALL LETTER I WITH DIAERESIS \\
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	620 RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK \\
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	621 INVERTED QUESTION MARK
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	622
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	623 in iso-8859-1), this increases the probability that a utf-8-sig
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	624 encoding can be correctly guessed from the byte sequence. So here the
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	625 BOM is not used to be able to determine the byte order used for
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	626 generating the byte sequence, but as a signature that helps in
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	627 guessing the encoding. On encoding the utf-8-sig codec will write
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	628 \code{0xef}, \code{0xbb}, \code{0xbf} as the first three bytes to the file.
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	629 On decoding utf-8-sig will skip those three bytes if they appear as the
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	630 first three bytes in the file.
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	631
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	632
32458 b73ab309ada7 link to the codecs page from the "".encode() description. Skip Montanaro <skip@pobox.com> parents: 31687 diff changeset	633 \subsection{Standard Encodings\label{standard-encodings}}
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	634
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	635 Python comes with a number of codecs builtin, either implemented as C
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	636 functions, or with dictionaries as mapping tables. The following table
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	637 lists the codecs by name, together with a few common aliases, and the
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	638 languages for which the encoding is likely used. Neither the list of
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	639 aliases nor the list of languages is meant to be exhaustive. Notice
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	640 that spelling alternatives that only differ in case or use a hyphen
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	641 instead of an underscore are also valid aliases.
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	642
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	643 Many of the character sets support the same languages. They vary in
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	644 individual characters (e.g. whether the EURO SIGN is supported or
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	645 not), and in the assignment of characters to code positions. For the
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	646 European languages in particular, the following variants typically
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	647 exist:
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	648
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	649 \begin{itemize}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	650 \item an ISO 8859 codeset
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	651 \item a Microsoft Windows code page, which is typically derived from
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	652 a 8859 codeset, but replaces control characters with additional
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	653 graphic characters
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	654 \item an IBM EBCDIC code page
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	655 \item an IBM PC code page, which is \ASCII{} compatible
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	656 \end{itemize}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	657
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	658 \begin{longtableiii}{l\|l\|l}{textrm}{Codec}{Aliases}{Languages}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	659
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	660 \lineiii{ascii}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	661 {646, us-ascii}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	662 {English}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	663
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	664 \lineiii{big5}
32722 4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	665 {big5-tw, csbig5}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	666 {Traditional Chinese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	667
32727 69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	668 \lineiii{big5hkscs}
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	669 {big5-hkscs, hkscs}
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	670 {Traditional Chinese}
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	671
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	672 \lineiii{cp037}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	673 {IBM037, IBM039}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	674 {English}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	675
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	676 \lineiii{cp424}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	677 {EBCDIC-CP-HE, IBM424}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	678 {Hebrew}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	679
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	680 \lineiii{cp437}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	681 {437, IBM437}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	682 {English}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	683
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	684 \lineiii{cp500}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	685 {EBCDIC-CP-BE, EBCDIC-CP-CH, IBM500}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	686 {Western Europe}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	687
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	688 \lineiii{cp737}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	689 {}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	690 {Greek}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	691
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	692 \lineiii{cp775}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	693 {IBM775}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	694 {Baltic languages}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	695
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	696 \lineiii{cp850}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	697 {850, IBM850}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	698 {Western Europe}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	699
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	700 \lineiii{cp852}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	701 {852, IBM852}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	702 {Central and Eastern Europe}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	703
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	704 \lineiii{cp855}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	705 {855, IBM855}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	706 {Bulgarian, Byelorussian, Macedonian, Russian, Serbian}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	707
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	708 \lineiii{cp856}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	709 {}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	710 {Hebrew}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	711
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	712 \lineiii{cp857}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	713 {857, IBM857}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	714 {Turkish}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	715
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	716 \lineiii{cp860}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	717 {860, IBM860}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	718 {Portuguese}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	719
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	720 \lineiii{cp861}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	721 {861, CP-IS, IBM861}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	722 {Icelandic}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	723
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	724 \lineiii{cp862}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	725 {862, IBM862}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	726 {Hebrew}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	727
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	728 \lineiii{cp863}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	729 {863, IBM863}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	730 {Canadian}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	731
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	732 \lineiii{cp864}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	733 {IBM864}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	734 {Arabic}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	735
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	736 \lineiii{cp865}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	737 {865, IBM865}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	738 {Danish, Norwegian}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	739
32462 1b9da5fe9053 add cp866 row Skip Montanaro <skip@pobox.com> parents: 32458 diff changeset	740 \lineiii{cp866}
1b9da5fe9053 add cp866 row Skip Montanaro <skip@pobox.com> parents: 32458 diff changeset	741 {866, IBM866}
1b9da5fe9053 add cp866 row Skip Montanaro <skip@pobox.com> parents: 32458 diff changeset	742 {Russian}
1b9da5fe9053 add cp866 row Skip Montanaro <skip@pobox.com> parents: 32458 diff changeset	743
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	744 \lineiii{cp869}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	745 {869, CP-GR, IBM869}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	746 {Greek}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	747
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	748 \lineiii{cp874}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	749 {}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	750 {Thai}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	751
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	752 \lineiii{cp875}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	753 {}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	754 {Greek}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	755
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	756 \lineiii{cp932}
32722 4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	757 {932, ms932, mskanji, ms-kanji}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	758 {Japanese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	759
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	760 \lineiii{cp949}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	761 {949, ms949, uhc}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	762 {Korean}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	763
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	764 \lineiii{cp950}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	765 {950, ms950}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	766 {Traditional Chinese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	767
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	768 \lineiii{cp1006}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	769 {}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	770 {Urdu}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	771
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	772 \lineiii{cp1026}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	773 {ibm1026}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	774 {Turkish}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	775
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	776 \lineiii{cp1140}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	777 {ibm1140}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	778 {Western Europe}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	779
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	780 \lineiii{cp1250}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	781 {windows-1250}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	782 {Central and Eastern Europe}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	783
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	784 \lineiii{cp1251}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	785 {windows-1251}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	786 {Bulgarian, Byelorussian, Macedonian, Russian, Serbian}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	787
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	788 \lineiii{cp1252}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	789 {windows-1252}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	790 {Western Europe}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	791
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	792 \lineiii{cp1253}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	793 {windows-1253}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	794 {Greek}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	795
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	796 \lineiii{cp1254}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	797 {windows-1254}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	798 {Turkish}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	799
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	800 \lineiii{cp1255}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	801 {windows-1255}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	802 {Hebrew}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	803
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	804 \lineiii{cp1256}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	805 {windows1256}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	806 {Arabic}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	807
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	808 \lineiii{cp1257}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	809 {windows-1257}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	810 {Baltic languages}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	811
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	812 \lineiii{cp1258}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	813 {windows-1258}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	814 {Vietnamese}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	815
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	816 \lineiii{euc_jp}
32722 4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	817 {eucjp, ujis, u-jis}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	818 {Japanese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	819
32727 69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	820 \lineiii{euc_jis_2004}
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	821 {jisx0213, eucjis2004}
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	822 {Japanese}
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	823
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	824 \lineiii{euc_jisx0213}
32727 69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	825 {eucjisx0213}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	826 {Japanese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	827
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	828 \lineiii{euc_kr}
32722 4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	829 {euckr, korean, ksc5601, ks_c-5601, ks_c-5601-1987, ksx1001, ks_x-1001}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	830 {Korean}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	831
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	832 \lineiii{gb2312}
32722 4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	833 {chinese, csiso58gb231280, euc-cn, euccn, eucgb2312-cn, gb2312-1980,
4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	834 gb2312-80, iso-ir-58}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	835 {Simplified Chinese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	836
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	837 \lineiii{gbk}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	838 {936, cp936, ms936}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	839 {Unified Chinese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	840
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	841 \lineiii{gb18030}
32722 4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	842 {gb18030-2000}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	843 {Unified Chinese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	844
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	845 \lineiii{hz}
32722 4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	846 {hzgb, hz-gb, hz-gb-2312}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	847 {Simplified Chinese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	848
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	849 \lineiii{iso2022_jp}
32722 4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	850 {csiso2022jp, iso2022jp, iso-2022-jp}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	851 {Japanese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	852
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	853 \lineiii{iso2022_jp_1}
32722 4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	854 {iso2022jp-1, iso-2022-jp-1}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	855 {Japanese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	856
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	857 \lineiii{iso2022_jp_2}
32722 4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	858 {iso2022jp-2, iso-2022-jp-2}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	859 {Japanese, Korean, Simplified Chinese, Western Europe, Greek}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	860
32727 69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	861 \lineiii{iso2022_jp_2004}
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	862 {iso2022jp-2004, iso-2022-jp-2004}
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	863 {Japanese}
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	864
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	865 \lineiii{iso2022_jp_3}
32722 4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	866 {iso2022jp-3, iso-2022-jp-3}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	867 {Japanese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	868
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	869 \lineiii{iso2022_jp_ext}
32722 4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	870 {iso2022jp-ext, iso-2022-jp-ext}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	871 {Japanese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	872
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	873 \lineiii{iso2022_kr}
32722 4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	874 {csiso2022kr, iso2022kr, iso-2022-kr}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	875 {Korean}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	876
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	877 \lineiii{latin_1}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	878 {iso-8859-1, iso8859-1, 8859, cp819, latin, latin1, L1}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	879 {West Europe}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	880
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	881 \lineiii{iso8859_2}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	882 {iso-8859-2, latin2, L2}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	883 {Central and Eastern Europe}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	884
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	885 \lineiii{iso8859_3}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	886 {iso-8859-3, latin3, L3}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	887 {Esperanto, Maltese}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	888
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	889 \lineiii{iso8859_4}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	890 {iso-8859-4, latin4, L4}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	891 {Baltic languagues}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	892
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	893 \lineiii{iso8859_5}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	894 {iso-8859-5, cyrillic}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	895 {Bulgarian, Byelorussian, Macedonian, Russian, Serbian}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	896
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	897 \lineiii{iso8859_6}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	898 {iso-8859-6, arabic}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	899 {Arabic}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	900
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	901 \lineiii{iso8859_7}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	902 {iso-8859-7, greek, greek8}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	903 {Greek}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	904
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	905 \lineiii{iso8859_8}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	906 {iso-8859-8, hebrew}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	907 {Hebrew}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	908
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	909 \lineiii{iso8859_9}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	910 {iso-8859-9, latin5, L5}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	911 {Turkish}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	912
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	913 \lineiii{iso8859_10}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	914 {iso-8859-10, latin6, L6}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	915 {Nordic languages}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	916
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	917 \lineiii{iso8859_13}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	918 {iso-8859-13}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	919 {Baltic languages}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	920
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	921 \lineiii{iso8859_14}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	922 {iso-8859-14, latin8, L8}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	923 {Celtic languages}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	924
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	925 \lineiii{iso8859_15}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	926 {iso-8859-15}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	927 {Western Europe}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	928
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	929 \lineiii{johab}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	930 {cp1361, ms1361}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	931 {Korean}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	932
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	933 \lineiii{koi8_r}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	934 {}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	935 {Russian}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	936
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	937 \lineiii{koi8_u}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	938 {}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	939 {Ukrainian}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	940
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	941 \lineiii{mac_cyrillic}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	942 {maccyrillic}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	943 {Bulgarian, Byelorussian, Macedonian, Russian, Serbian}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	944
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	945 \lineiii{mac_greek}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	946 {macgreek}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	947 {Greek}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	948
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	949 \lineiii{mac_iceland}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	950 {maciceland}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	951 {Icelandic}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	952
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	953 \lineiii{mac_latin2}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	954 {maclatin2, maccentraleurope}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	955 {Central and Eastern Europe}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	956
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	957 \lineiii{mac_roman}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	958 {macroman}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	959 {Western Europe}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	960
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	961 \lineiii{mac_turkish}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	962 {macturkish}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	963 {Turkish}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	964
31687 a885e4d64ad0 Add a new unicode codec: ptcp154 (Kazakh) Hye-Shik Chang <hyeshik@gmail.com> parents: 31386 diff changeset	965 \lineiii{ptcp154}
a885e4d64ad0 Add a new unicode codec: ptcp154 (Kazakh) Hye-Shik Chang <hyeshik@gmail.com> parents: 31386 diff changeset	966 {csptcp154, pt154, cp154, cyrillic-asian}
a885e4d64ad0 Add a new unicode codec: ptcp154 (Kazakh) Hye-Shik Chang <hyeshik@gmail.com> parents: 31386 diff changeset	967 {Kazakh}
a885e4d64ad0 Add a new unicode codec: ptcp154 (Kazakh) Hye-Shik Chang <hyeshik@gmail.com> parents: 31386 diff changeset	968
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	969 \lineiii{shift_jis}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	970 {csshiftjis, shiftjis, sjis, s_jis}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	971 {Japanese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	972
32727 69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	973 \lineiii{shift_jis_2004}
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	974 {shiftjis2004, sjis_2004, sjis2004}
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	975 {Japanese}
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	976
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	977 \lineiii{shift_jisx0213}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	978 {shiftjisx0213, sjisx0213, s_jisx0213}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	979 {Japanese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	980
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	981 \lineiii{utf_16}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	982 {U16, utf16}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	983 {all languages}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	984
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	985 \lineiii{utf_16_be}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	986 {UTF-16BE}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	987 {all languages (BMP only)}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	988
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	989 \lineiii{utf_16_le}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	990 {UTF-16LE}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	991 {all languages (BMP only)}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	992
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	993 \lineiii{utf_7}
35710 59c4a65642c5 Bug #1245379: Add "unicode-1-1-utf-7" as an alias for "utf-7" as specified Walter Dörwald <walter@livinglogic.de> parents: 35416 diff changeset	994 {U7, unicode-1-1-utf-7}
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	995 {all languages}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	996
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	997 \lineiii{utf_8}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	998 {U8, UTF, utf8}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	999 {all languages}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1000
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1001 \lineiii{utf_8_sig}
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1002 {}
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1003 {all languages}
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1004
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1005 \end{longtableiii}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1006
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1007 A number of codecs are specific to Python, so their codec names have
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1008 no meaning outside Python. Some of them don't convert from Unicode
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1009 strings to byte strings, but instead use the property of the Python
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1010 codecs machinery that any bijective function with one argument can be
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1011 considered as an encoding.
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1012
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1013 For the codecs listed below, the result in the ``encoding'' direction
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1014 is always a byte string. The result of the ``decoding'' direction is
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1015 listed as operand type in the table.
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1016
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1017 \begin{tableiv}{l\|l\|l\|l}{textrm}{Codec}{Aliases}{Operand type}{Purpose}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1018
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1019 \lineiv{base64_codec}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1020 {base64, base-64}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1021 {byte string}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1022 {Convert operand to MIME base64}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1023
30522 db63cb0fc7da Added codec for bz2 compression. Raymond Hettinger <python@rcn.com> parents: 30505 diff changeset	1024 \lineiv{bz2_codec}
db63cb0fc7da Added codec for bz2 compression. Raymond Hettinger <python@rcn.com> parents: 30505 diff changeset	1025 {bz2}
db63cb0fc7da Added codec for bz2 compression. Raymond Hettinger <python@rcn.com> parents: 30505 diff changeset	1026 {byte string}
db63cb0fc7da Added codec for bz2 compression. Raymond Hettinger <python@rcn.com> parents: 30505 diff changeset	1027 {Compress the operand using bz2}
db63cb0fc7da Added codec for bz2 compression. Raymond Hettinger <python@rcn.com> parents: 30505 diff changeset	1028
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1029 \lineiv{hex_codec}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1030 {hex}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1031 {byte string}
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1032 {Convert operand to hexadecimal representation, with two
1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1033 digits per byte}
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1034
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1035 \lineiv{idna}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1036 {}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1037 {Unicode string}
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1038 {Implements \rfc{3490}.
30273 244b55c90e0f Minor typo Raymond Hettinger <python@rcn.com> parents: 30134 diff changeset	1039 \versionadded{2.3}
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1040 See also \refmodule{encodings.idna}}
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1041
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1042 \lineiv{mbcs}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1043 {dbcs}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1044 {Unicode string}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1045 {Windows only: Encode operand according to the ANSI codepage (CP_ACP)}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1046
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1047 \lineiv{palmos}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1048 {}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1049 {Unicode string}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1050 {Encoding of PalmOS 3.5}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1051
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1052 \lineiv{punycode}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1053 {}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1054 {Unicode string}
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1055 {Implements \rfc{3492}.
1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1056 \versionadded{2.3}}
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1057
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1058 \lineiv{quopri_codec}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1059 {quopri, quoted-printable, quotedprintable}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1060 {byte string}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1061 {Convert operand to MIME quoted printable}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1062
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1063 \lineiv{raw_unicode_escape}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1064 {}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1065 {Unicode string}
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1066 {Produce a string that is suitable as raw Unicode literal in
1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1067 Python source code}
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1068
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1069 \lineiv{rot_13}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1070 {rot13}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1071 {byte string}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1072 {Returns the Caesar-cypher encryption of the operand}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1073
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1074 \lineiv{string_escape}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1075 {}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1076 {byte string}
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1077 {Produce a string that is suitable as string literal in
1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1078 Python source code}
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1079
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1080 \lineiv{undefined}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1081 {}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1082 {any}
36913 0e36e7c66b8e typo Georg Brandl <georg@python.org> parents: 36637 diff changeset	1083 {Raise an exception for all conversions. Can be used as the
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1084 system encoding if no automatic coercion between byte and
1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1085 Unicode strings is desired.}
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1086
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1087 \lineiv{unicode_escape}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1088 {}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1089 {Unicode string}
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1090 {Produce a string that is suitable as Unicode literal in
1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1091 Python source code}
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1092
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1093 \lineiv{unicode_internal}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1094 {}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1095 {Unicode string}
34296 67f2190e1f6d SF Patch #1093896: miscellaneous doc typos Raymond Hettinger <python@rcn.com> parents: 33437 diff changeset	1096 {Return the internal representation of the operand}
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1097
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1098 \lineiv{uu_codec}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1099 {uu}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1100 {byte string}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1101 {Convert the operand using uuencode}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1102
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1103 \lineiv{zlib_codec}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1104 {zip, zlib}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1105 {byte string}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1106 {Compress the operand using gzip}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1107
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1108 \end{tableiv}
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1109
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1110 \subsection{\module{encodings.idna} ---
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1111 Internationalized Domain Names in Applications}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1112
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1113 \declaremodule{standard}{encodings.idna}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1114 \modulesynopsis{Internationalized Domain Names implementation}
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1115 % XXX The next line triggers a formatting bug, so it's commented out
1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1116 % until that can be fixed.
1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1117 %\moduleauthor{Martin v. L\"owis}
1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1118
1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1119 \versionadded{2.3}
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1120
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1121 This module implements \rfc{3490} (Internationalized Domain Names in
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1122 Applications) and \rfc{3492} (Nameprep: A Stringprep Profile for
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1123 Internationalized Domain Names (IDN)). It builds upon the
29894 ac84f179bb84 A variety of markup-level adjustments. Fred Drake <fdrake@acm.org> parents: 29528 diff changeset	1124 \code{punycode} encoding and \refmodule{stringprep}.
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1125
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1126 These RFCs together define a protocol to support non-\ASCII{} characters
1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1127 in domain names. A domain name containing non-\ASCII{} characters (such
29894 ac84f179bb84 A variety of markup-level adjustments. Fred Drake <fdrake@acm.org> parents: 29528 diff changeset	1128 as ``www.Alliancefran\c caise.nu'') is converted into an
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1129 \ASCII-compatible encoding (ACE, such as
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1130 ``www.xn--alliancefranaise-npb.nu''). The ACE form of the domain name
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1131 is then used in all places where arbitrary characters are not allowed
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1132 by the protocol, such as DNS queries, HTTP \mailheader{Host} fields, and so
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1133 on. This conversion is carried out in the application; if possible
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1134 invisible to the user: The application should transparently convert
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1135 Unicode domain labels to IDNA on the wire, and convert back ACE labels
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1136 to Unicode before presenting them to the user.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1137
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1138 Python supports this conversion in several ways: The \code{idna} codec
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1139 allows to convert between Unicode and the ACE. Furthermore, the
29894 ac84f179bb84 A variety of markup-level adjustments. Fred Drake <fdrake@acm.org> parents: 29528 diff changeset	1140 \refmodule{socket} module transparently converts Unicode host names to
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1141 ACE, so that applications need not be concerned about converting host
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1142 names themselves when they pass them to the socket module. On top of
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1143 that, modules that have host names as function parameters, such as
29894 ac84f179bb84 A variety of markup-level adjustments. Fred Drake <fdrake@acm.org> parents: 29528 diff changeset	1144 \refmodule{httplib} and \refmodule{ftplib}, accept Unicode host names
ac84f179bb84 A variety of markup-level adjustments. Fred Drake <fdrake@acm.org> parents: 29528 diff changeset	1145 (\refmodule{httplib} then also transparently sends an IDNA hostname in
ac84f179bb84 A variety of markup-level adjustments. Fred Drake <fdrake@acm.org> parents: 29528 diff changeset	1146 the \mailheader{Host} field if it sends that field at all).
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1147
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1148 When receiving host names from the wire (such as in reverse name
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1149 lookup), no automatic conversion to Unicode is performed: Applications
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1150 wishing to present such host names to the user should decode them to
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1151 Unicode.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1152
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1153 The module \module{encodings.idna} also implements the nameprep
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1154 procedure, which performs certain normalizations on host names, to
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1155 achieve case-insensitivity of international domain names, and to unify
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1156 similar characters. The nameprep functions can be used directly if
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1157 desired.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1158
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1159 \begin{funcdesc}{nameprep}{label}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1160 Return the nameprepped version of \var{label}. The implementation
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1161 currently assumes query strings, so \code{AllowUnassigned} is
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1162 true.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1163 \end{funcdesc}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1164
29528 7aee6ef596dd Fix typo. Raymond Hettinger <python@rcn.com> parents: 28906 diff changeset	1165 \begin{funcdesc}{ToASCII}{label}
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1166 Convert a label to \ASCII, as specified in \rfc{3490}.
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1167 \code{UseSTD3ASCIIRules} is assumed to be false.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1168 \end{funcdesc}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1169
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1170 \begin{funcdesc}{ToUnicode}{label}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1171 Convert a label to Unicode, as specified in \rfc{3490}.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1172 \end{funcdesc}
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1173
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1174 \subsection{\module{encodings.utf_8_sig} ---
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1175 UTF-8 codec with BOM signature}
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1176 \declaremodule{standard}{encodings.utf-8-sig} % XXX utf_8_sig gives TeX errors
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1177 \modulesynopsis{UTF-8 codec with BOM signature}
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1178 \moduleauthor{Walter D\"orwald}
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1179
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1180 \versionadded{2.5}
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1181
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1182 This module implements a variant of the UTF-8 codec: On encoding a
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1183 UTF-8 encoded BOM will be prepended to the UTF-8 encoded bytes. For
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1184 the stateful encoder this is only done once (on the first write to the
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1185 byte stream). For decoding an optional UTF-8 encoded BOM at the start
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1186 of the data will be skipped.

Mercurial > cpython-lru_cache-weakref

annotate Doc/lib/libcodecs.tex @ 37850:ebe59cd18273