cpython-lru_cache-weakref: Doc/lib/libcodecs.tex annotate

annotate Doc/lib/libcodecs.tex @ 39042:c02fa187e0f4 legacy-trunk

Part of SF patch #1484695. This removes dead code. The chksum was already verified in .frombuf() on the lines above. If there was a problem an exception is raised, so there was no way this condition could have been true.

author	Neal Norwitz <nnorwitz@gmail.com>
date	Mon, 10 Jul 2006 00:23:17 +0000
parents	52551221a132
children	ea6f2c5d2749

rev	line source
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	1 \section{\module{codecs} ---
12274 559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	2 Codec registry and base classes}
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	3
12274 559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	4 \declaremodule{standard}{codecs}
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	5 \modulesynopsis{Encode and decode data and streams.}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	6 \moduleauthor{Marc-Andre Lemburg}{mal@lemburg.com}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	7 \sectionauthor{Marc-Andre Lemburg}{mal@lemburg.com}
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	8 \sectionauthor{Martin v. L\"owis}{martin@v.loewis.de}
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	9
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	10 \index{Unicode}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	11 \index{Codecs}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	12 \indexii{Codecs}{encode}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	13 \indexii{Codecs}{decode}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	14 \index{streams}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	15 \indexii{stackable}{streams}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	16
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	17
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	18 This module defines base classes for standard Python codecs (encoders
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	19 and decoders) and provides access to the internal Python codec
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	20 registry which manages the codec and error handling lookup process.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	21
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	22 It defines the following functions:
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	23
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	24 \begin{funcdesc}{register}{search_function}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	25 Register a codec search function. Search functions are expected to
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	26 take one argument, the encoding name in all lower case letters, and
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	27 return a \class{CodecInfo} object having the following attributes:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	28
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	29 \begin{itemize}
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	30 \item \code{name} The name of the encoding;
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	31 \item \code{encoder} The stateless encoding function;
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	32 \item \code{decoder} The stateless decoding function;
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	33 \item \code{incrementalencoder} An incremental encoder class or factory function;
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	34 \item \code{incrementaldecoder} An incremental decoder class or factory function;
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	35 \item \code{streamwriter} A stream writer class or factory function;
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	36 \item \code{streamreader} A stream reader class or factory function.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	37 \end{itemize}
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	38
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	39 The various functions or classes take the following arguments:
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	40
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	41 \var{encoder} and \var{decoder}: These must be functions or methods
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	42 which have the same interface as the
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	43 \method{encode()}/\method{decode()} methods of Codec instances (see
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	44 Codec Interface). The functions/methods are expected to work in a
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	45 stateless mode.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	46
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	47 \var{incrementalencoder} and \var{incrementalencoder}: These have to be
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	48 factory functions providing the following interface:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	49
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	50 \code{factory(\var{errors}='strict')}
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	51
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	52 The factory functions must return objects providing the interfaces
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	53 defined by the base classes \class{IncrementalEncoder} and
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	54 \class{IncrementalEncoder}, respectively. Incremental codecs can maintain
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	55 state.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	56
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	57 \var{streamreader} and \var{streamwriter}: These have to be
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	58 factory functions providing the following interface:
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	59
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	60 \code{factory(\var{stream}, \var{errors}='strict')}
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	61
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	62 The factory functions must return objects providing the interfaces
12274 559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	63 defined by the base classes \class{StreamWriter} and
559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	64 \class{StreamReader}, respectively. Stream codecs can maintain
559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	65 state.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	66
12274 559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	67 Possible values for errors are \code{'strict'} (raise an exception
559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	68 in case of an encoding error), \code{'replace'} (replace malformed
26294 6d0cd80e2804 Document additional error handling names available through PEP 293. Walter Dörwald <walter@livinglogic.de> parents: 26099 diff changeset	69 data with a suitable replacement marker, such as \character{?}),
12274 559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	70 \code{'ignore'} (ignore malformed data and continue without further
26294 6d0cd80e2804 Document additional error handling names available through PEP 293. Walter Dörwald <walter@livinglogic.de> parents: 26099 diff changeset	71 notice), \code{'xmlcharrefreplace'} (replace with the appropriate XML
6d0cd80e2804 Document additional error handling names available through PEP 293. Walter Dörwald <walter@livinglogic.de> parents: 26099 diff changeset	72 character reference (for encoding only)) and \code{'backslashreplace'}
6d0cd80e2804 Document additional error handling names available through PEP 293. Walter Dörwald <walter@livinglogic.de> parents: 26099 diff changeset	73 (replace with backslashed escape sequences (for encoding only)) as
6d0cd80e2804 Document additional error handling names available through PEP 293. Walter Dörwald <walter@livinglogic.de> parents: 26099 diff changeset	74 well as any other error handling name defined via
6d0cd80e2804 Document additional error handling names available through PEP 293. Walter Dörwald <walter@livinglogic.de> parents: 26099 diff changeset	75 \function{register_error()}.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	76
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	77 In case a search function cannot find a given encoding, it should
12274 559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	78 return \code{None}.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	79 \end{funcdesc}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	80
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	81 \begin{funcdesc}{lookup}{encoding}
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	82 Looks up the codec info in the Python codec registry and returns a
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	83 \class{CodecInfo} object as defined above.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	84
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	85 Encodings are first looked up in the registry's cache. If not found,
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	86 the list of registered search functions is scanned. If no \class{CodecInfo}
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	87 object is found, a \exception{LookupError} is raised. Otherwise, the
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	88 \class{CodecInfo} object is stored in the cache and returned to the caller.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	89 \end{funcdesc}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	90
23112 a3c982d412b4 typo Skip Montanaro <skip@pobox.com> parents: 23087 diff changeset	91 To simplify access to the various codecs, the module provides these
19982 bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	92 additional functions which use \function{lookup()} for the codec
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	93 lookup:
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	94
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	95 \begin{funcdesc}{getencoder}{encoding}
37884 b49843a3a1df Typo fixes Andrew M. Kuchling <amk@amk.ca> parents: 37522 diff changeset	96 Look up the codec for the given encoding and return its encoder
19982 bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	97 function.
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	98
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	99 Raises a \exception{LookupError} in case the encoding cannot be found.
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	100 \end{funcdesc}
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	101
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	102 \begin{funcdesc}{getdecoder}{encoding}
37884 b49843a3a1df Typo fixes Andrew M. Kuchling <amk@amk.ca> parents: 37522 diff changeset	103 Look up the codec for the given encoding and return its decoder
19982 bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	104 function.
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	105
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	106 Raises a \exception{LookupError} in case the encoding cannot be found.
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	107 \end{funcdesc}
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	108
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	109 \begin{funcdesc}{getincrementalencoder}{encoding}
37884 b49843a3a1df Typo fixes Andrew M. Kuchling <amk@amk.ca> parents: 37522 diff changeset	110 Look up the codec for the given encoding and return its incremental encoder
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	111 class or factory function.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	112
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	113 Raises a \exception{LookupError} in case the encoding cannot be found or the
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	114 codec doesn't support an incremental encoder.
37024 f70c036489ec Add some versionadded info to new incremental codec docs and fix doco nits. Neal Norwitz <nnorwitz@gmail.com> parents: 36996 diff changeset	115 \versionadded{2.5}
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	116 \end{funcdesc}
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	117
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	118 \begin{funcdesc}{getincrementaldecoder}{encoding}
37884 b49843a3a1df Typo fixes Andrew M. Kuchling <amk@amk.ca> parents: 37522 diff changeset	119 Look up the codec for the given encoding and return its incremental decoder
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	120 class or factory function.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	121
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	122 Raises a \exception{LookupError} in case the encoding cannot be found or the
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	123 codec doesn't support an incremental decoder.
37024 f70c036489ec Add some versionadded info to new incremental codec docs and fix doco nits. Neal Norwitz <nnorwitz@gmail.com> parents: 36996 diff changeset	124 \versionadded{2.5}
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	125 \end{funcdesc}
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	126
19982 bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	127 \begin{funcdesc}{getreader}{encoding}
37884 b49843a3a1df Typo fixes Andrew M. Kuchling <amk@amk.ca> parents: 37522 diff changeset	128 Look up the codec for the given encoding and return its StreamReader
19982 bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	129 class or factory function.
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	130
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	131 Raises a \exception{LookupError} in case the encoding cannot be found.
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	132 \end{funcdesc}
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	133
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	134 \begin{funcdesc}{getwriter}{encoding}
37884 b49843a3a1df Typo fixes Andrew M. Kuchling <amk@amk.ca> parents: 37522 diff changeset	135 Look up the codec for the given encoding and return its StreamWriter
19982 bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	136 class or factory function.
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	137
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	138 Raises a \exception{LookupError} in case the encoding cannot be found.
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	139 \end{funcdesc}
bf4b1b08a2c6 Docs and News item for the codecs.py additions. Marc-André Lemburg <mal@egenix.com> parents: 16127 diff changeset	140
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	141 \begin{funcdesc}{register_error}{name, error_handler}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	142 Register the error handling function \var{error_handler} under the
25172 5a17f0978326 Fix typo. Close SF Bug 606354. Raymond Hettinger <python@rcn.com> parents: 25091 diff changeset	143 name \var{name}. \var{error_handler} will be called during encoding
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	144 and decoding in case of an error, when \var{name} is specified as the
27555 c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	145 errors parameter.
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	146
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	147 For encoding \var{error_handler} will be called with a
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	148 \exception{UnicodeEncodeError} instance, which contains information about
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	149 the location of the error. The error handler must either raise this or
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	150 a different exception or return a tuple with a replacement for the
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	151 unencodable part of the input and a position where encoding should
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	152 continue. The encoder will encode the replacement and continue encoding
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	153 the original input at the specified position. Negative position values
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	154 will be treated as being relative to the end of the input string. If the
37064 64c83e0f683e More \exception fixes. Georg Brandl <georg@python.org> parents: 37024 diff changeset	155 resulting position is out of bound an \exception{IndexError} will be raised.
27555 c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	156
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	157 Decoding and translating works similar, except \exception{UnicodeDecodeError}
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	158 or \exception{UnicodeTranslateError} will be passed to the handler and
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	159 that the replacement from the error handler will be put into the output
c267f28724dd Change the treatment of positions returned by PEP293 Walter Dörwald <walter@livinglogic.de> parents: 26943 diff changeset	160 directly.
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	161 \end{funcdesc}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	162
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	163 \begin{funcdesc}{lookup_error}{name}
37902 52551221a132 Correct the grammar George Yoshida <dynkin@gmail.com> parents: 37899 diff changeset	164 Return the error handler previously registered under the name \var{name}.
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	165
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	166 Raises a \exception{LookupError} in case the handler cannot be found.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	167 \end{funcdesc}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	168
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	169 \begin{funcdesc}{strict_errors}{exception}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	170 Implements the \code{strict} error handling.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	171 \end{funcdesc}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	172
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	173 \begin{funcdesc}{replace_errors}{exception}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	174 Implements the \code{replace} error handling.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	175 \end{funcdesc}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	176
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	177 \begin{funcdesc}{ignore_errors}{exception}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	178 Implements the \code{ignore} error handling.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	179 \end{funcdesc}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	180
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	181 \begin{funcdesc}{xmlcharrefreplace_errors_errors}{exception}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	182 Implements the \code{xmlcharrefreplace} error handling.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	183 \end{funcdesc}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	184
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	185 \begin{funcdesc}{backslashreplace_errors_errors}{exception}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	186 Implements the \code{backslashreplace} error handling.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	187 \end{funcdesc}
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 23658 diff changeset	188
25989 611af79c0e89 Move introductory sentence to where it belongs. Walter Dörwald <walter@livinglogic.de> parents: 25172 diff changeset	189 To simplify working with encoded files or stream, the module
611af79c0e89 Move introductory sentence to where it belongs. Walter Dörwald <walter@livinglogic.de> parents: 25172 diff changeset	190 also defines these utility functions:
611af79c0e89 Move introductory sentence to where it belongs. Walter Dörwald <walter@livinglogic.de> parents: 25172 diff changeset	191
13696 bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	192 \begin{funcdesc}{open}{filename, mode\optional{, encoding\optional{,
bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	193 errors\optional{, buffering}}}}
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	194 Open an encoded file using the given \var{mode} and return
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	195 a wrapped version providing transparent encoding/decoding.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	196
20580 376ee174ecd0 Use the \note and \warning macros where appropriate. Fred Drake <fdrake@acm.org> parents: 19982 diff changeset	197 \note{The wrapped version will only accept the object format
13696 bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	198 defined by the codecs, i.e.\ Unicode objects for most built-in
bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	199 codecs. Output is also codec-dependent and will usually be Unicode as
20580 376ee174ecd0 Use the \note and \warning macros where appropriate. Fred Drake <fdrake@acm.org> parents: 19982 diff changeset	200 well.}
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	201
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	202 \var{encoding} specifies the encoding which is to be used for the
30505 f4a01c405a44 Fix doubled word errors. Raymond Hettinger <python@rcn.com> parents: 30273 diff changeset	203 file.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	204
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	205 \var{errors} may be given to define the error handling. It defaults
13696 bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	206 to \code{'strict'} which causes a \exception{ValueError} to be raised
bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	207 in case an encoding error occurs.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	208
12274 559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	209 \var{buffering} has the same meaning as for the built-in
559b25a79032 Make sure the \declaremodule uses the right name for the module! Fred Drake <fdrake@acm.org> parents: 12260 diff changeset	210 \function{open()} function. It defaults to line buffered.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	211 \end{funcdesc}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	212
13696 bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	213 \begin{funcdesc}{EncodedFile}{file, input\optional{,
bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	214 output\optional{, errors}}}
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	215 Return a wrapped version of file which provides transparent
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	216 encoding translation.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	217
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	218 Strings written to the wrapped file are interpreted according to the
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	219 given \var{input} encoding and then written to the original file as
13696 bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	220 strings using the \var{output} encoding. The intermediate encoding will
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	221 usually be Unicode but depends on the specified codecs.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	222
13696 bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	223 If \var{output} is not given, it defaults to \var{input}.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	224
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	225 \var{errors} may be given to define the error handling. It defaults to
13696 bb364d6f9f69 Fix small typos and markup consistency nits. Fred Drake <fdrake@acm.org> parents: 12274 diff changeset	226 \code{'strict'}, which causes \exception{ValueError} to be raised in case
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	227 an encoding error occurs.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	228 \end{funcdesc}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	229
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	230 \begin{funcdesc}{iterencode}{iterable, encoding\optional{, errors}}
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	231 Uses an incremental encoder to iteratively encode the input provided by
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	232 \var{iterable}. This function is a generator. \var{errors} (as well as
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	233 any other keyword argument) is passed through to the incremental encoder.
37024 f70c036489ec Add some versionadded info to new incremental codec docs and fix doco nits. Neal Norwitz <nnorwitz@gmail.com> parents: 36996 diff changeset	234 \versionadded{2.5}
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	235 \end{funcdesc}
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	236
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	237 \begin{funcdesc}{iterdecode}{iterable, encoding\optional{, errors}}
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	238 Uses an incremental decoder to iteratively decode the input provided by
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	239 \var{iterable}. This function is a generator. \var{errors} (as well as
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	240 any other keyword argument) is passed through to the incremental encoder.
37024 f70c036489ec Add some versionadded info to new incremental codec docs and fix doco nits. Neal Norwitz <nnorwitz@gmail.com> parents: 36996 diff changeset	241 \versionadded{2.5}
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	242 \end{funcdesc}
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	243
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	244 The module also provides the following constants which are useful
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	245 for reading and writing to platform dependent files:
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	246
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	247 \begin{datadesc}{BOM}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	248 \dataline{BOM_BE}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	249 \dataline{BOM_LE}
23658 94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	250 \dataline{BOM_UTF8}
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	251 \dataline{BOM_UTF16}
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	252 \dataline{BOM_UTF16_BE}
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	253 \dataline{BOM_UTF16_LE}
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	254 \dataline{BOM_UTF32}
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	255 \dataline{BOM_UTF32_BE}
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	256 \dataline{BOM_UTF32_LE}
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	257 These constants define various encodings of the Unicode byte order mark
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	258 (BOM) used in UTF-16 and UTF-32 data streams to indicate the byte order
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	259 used in the stream or file and in UTF-8 as a Unicode signature.
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	260 \constant{BOM_UTF16} is either \constant{BOM_UTF16_BE} or
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	261 \constant{BOM_UTF16_LE} depending on the platform's native byte order,
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	262 \constant{BOM} is an alias for \constant{BOM_UTF16}, \constant{BOM_LE}
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	263 for \constant{BOM_UTF16_LE} and \constant{BOM_BE} for \constant{BOM_UTF16_BE}.
94aa4eb9bb7d Add constants BOM_UTF8, BOM_UTF16, BOM_UTF16_LE, BOM_UTF16_BE, Walter Dörwald <walter@livinglogic.de> parents: 23112 diff changeset	264 The others represent the BOM in UTF-8 and UTF-32 encodings.
12260 8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	265 \end{datadesc}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: diff changeset	266
16127 6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	267
35951 90797233208e SF patch #1364946: Add a reference link from the dcoumentation of the encode Walter Dörwald <walter@livinglogic.de> parents: 35739 diff changeset	268 \subsection{Codec Base Classes \label{codec-base-classes}}
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	269
35739 d0399379d0cb add missing word Fred Drake <fdrake@acm.org> parents: 35710 diff changeset	270 The \module{codecs} module defines a set of base classes which define the
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	271 interface and can also be used to easily write you own codecs for use
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	272 in Python.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	273
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	274 Each codec has to define four interfaces to make it usable as codec in
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	275 Python: stateless encoder, stateless decoder, stream reader and stream
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	276 writer. The stream reader and writers typically reuse the stateless
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	277 encoder/decoder to implement the file protocols.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	278
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	279 The \class{Codec} class defines the interface for stateless
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	280 encoders/decoders.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	281
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	282 To simplify and standardize error handling, the \method{encode()} and
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	283 \method{decode()} methods may implement different error handling
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	284 schemes by providing the \var{errors} string argument. The following
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	285 string values are defined and implemented by all standard Python
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	286 codecs:
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	287
16127 6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	288 \begin{tableii}{l\|l}{code}{Value}{Meaning}
26099 135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	289 \lineii{'strict'}{Raise \exception{UnicodeError} (or a subclass);
16127 6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	290 this is the default.}
6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	291 \lineii{'ignore'}{Ignore the character and continue with the next.}
6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	292 \lineii{'replace'}{Replace with a suitable replacement character;
6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	293 Python will use the official U+FFFD REPLACEMENT
26099 135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	294 CHARACTER for the built-in Unicode codecs on
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	295 decoding and '?' on encoding.}
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	296 \lineii{'xmlcharrefreplace'}{Replace with the appropriate XML
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	297 character reference (only for encoding).}
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	298 \lineii{'backslashreplace'}{Replace with backslashed escape sequences
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	299 (only for encoding).}
16127 6f1fd8500398 Added link to the "Python Codecs" project at SourceForge. Fred Drake <fdrake@acm.org> parents: 15186 diff changeset	300 \end{tableii}
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	301
26099 135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	302 The set of allowed values can be extended via \method{register_error}.
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	303
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	304
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	305 \subsubsection{Codec Objects \label{codec-objects}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	306
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	307 The \class{Codec} class defines these methods which also define the
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	308 function interfaces of the stateless encoder and decoder:
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	309
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	310 \begin{methoddesc}{encode}{input\optional{, errors}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	311 Encodes the object \var{input} and returns a tuple (output object,
23087 09db3f5d0bd0 added small clarification to the descriptions of encode() and decode() Skip Montanaro <skip@pobox.com> parents: 20580 diff changeset	312 length consumed). While codecs are not restricted to use with Unicode, in
09db3f5d0bd0 added small clarification to the descriptions of encode() and decode() Skip Montanaro <skip@pobox.com> parents: 20580 diff changeset	313 a Unicode context, encoding converts a Unicode object to a plain string
09db3f5d0bd0 added small clarification to the descriptions of encode() and decode() Skip Montanaro <skip@pobox.com> parents: 20580 diff changeset	314 using a particular character set encoding (e.g., \code{cp1252} or
09db3f5d0bd0 added small clarification to the descriptions of encode() and decode() Skip Montanaro <skip@pobox.com> parents: 20580 diff changeset	315 \code{iso-8859-1}).
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	316
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	317 \var{errors} defines the error handling to apply. It defaults to
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	318 \code{'strict'} handling.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	319
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	320 The method may not store state in the \class{Codec} instance. Use
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	321 \class{StreamCodec} for codecs which have to keep state in order to
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	322 make encoding/decoding efficient.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	323
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	324 The encoder must be able to handle zero length input and return an
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	325 empty object of the output object type in this situation.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	326 \end{methoddesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	327
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	328 \begin{methoddesc}{decode}{input\optional{, errors}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	329 Decodes the object \var{input} and returns a tuple (output object,
23087 09db3f5d0bd0 added small clarification to the descriptions of encode() and decode() Skip Montanaro <skip@pobox.com> parents: 20580 diff changeset	330 length consumed). In a Unicode context, decoding converts a plain string
09db3f5d0bd0 added small clarification to the descriptions of encode() and decode() Skip Montanaro <skip@pobox.com> parents: 20580 diff changeset	331 encoded using a particular character set encoding to a Unicode object.
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	332
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	333 \var{input} must be an object which provides the \code{bf_getreadbuf}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	334 buffer slot. Python strings, buffer objects and memory mapped files
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	335 are examples of objects providing this slot.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	336
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	337 \var{errors} defines the error handling to apply. It defaults to
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	338 \code{'strict'} handling.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	339
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	340 The method may not store state in the \class{Codec} instance. Use
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	341 \class{StreamCodec} for codecs which have to keep state in order to
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	342 make encoding/decoding efficient.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	343
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	344 The decoder must be able to handle zero length input and return an
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	345 empty object of the output object type in this situation.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	346 \end{methoddesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	347
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	348 The \class{IncrementalEncoder} and \class{IncrementalDecoder} classes provide
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	349 the basic interface for incremental encoding and decoding. Encoding/decoding the
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	350 input isn't done with one call to the stateless encoder/decoder function,
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	351 but with multiple calls to the \method{encode}/\method{decode} method of the
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	352 incremental encoder/decoder. The incremental encoder/decoder keeps track of
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	353 the encoding/decoding process during method calls.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	354
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	355 The joined output of calls to the \method{encode}/\method{decode} method is the
37888 5bc915feed14 Typo, grammar fixes. This file could use another proofreading pass. Andrew M. Kuchling <amk@amk.ca> parents: 37884 diff changeset	356 same as if all the single inputs were joined into one, and this input was
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	357 encoded/decoded with the stateless encoder/decoder.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	358
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	359
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	360 \subsubsection{IncrementalEncoder Objects \label{incremental-encoder-objects}}
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	361
37024 f70c036489ec Add some versionadded info to new incremental codec docs and fix doco nits. Neal Norwitz <nnorwitz@gmail.com> parents: 36996 diff changeset	362 \versionadded{2.5}
f70c036489ec Add some versionadded info to new incremental codec docs and fix doco nits. Neal Norwitz <nnorwitz@gmail.com> parents: 36996 diff changeset	363
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	364 The \class{IncrementalEncoder} class is used for encoding an input in multiple
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	365 steps. It defines the following methods which every incremental encoder must
37888 5bc915feed14 Typo, grammar fixes. This file could use another proofreading pass. Andrew M. Kuchling <amk@amk.ca> parents: 37884 diff changeset	366 define in order to be compatible with the Python codec registry.
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	367
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	368 \begin{classdesc}{IncrementalEncoder}{\optional{errors}}
37902 52551221a132 Correct the grammar George Yoshida <dynkin@gmail.com> parents: 37899 diff changeset	369 Constructor for an \class{IncrementalEncoder} instance.
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	370
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	371 All incremental encoders must provide this constructor interface. They are
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	372 free to add additional keyword arguments, but only the ones defined
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	373 here are used by the Python codec registry.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	374
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	375 The \class{IncrementalEncoder} may implement different error handling
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	376 schemes by providing the \var{errors} keyword argument. These
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	377 parameters are predefined:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	378
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	379 \begin{itemize}
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	380 \item \code{'strict'} Raise \exception{ValueError} (or a subclass);
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	381 this is the default.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	382 \item \code{'ignore'} Ignore the character and continue with the next.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	383 \item \code{'replace'} Replace with a suitable replacement character
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	384 \item \code{'xmlcharrefreplace'} Replace with the appropriate XML
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	385 character reference
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	386 \item \code{'backslashreplace'} Replace with backslashed escape sequences.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	387 \end{itemize}
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	388
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	389 The \var{errors} argument will be assigned to an attribute of the
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	390 same name. Assigning to this attribute makes it possible to switch
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	391 between different error handling strategies during the lifetime
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	392 of the \class{IncrementalEncoder} object.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	393
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	394 The set of allowed values for the \var{errors} argument can
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	395 be extended with \function{register_error()}.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	396 \end{classdesc}
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	397
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	398 \begin{methoddesc}{encode}{object\optional{, final}}
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	399 Encodes \var{object} (taking the current state of the encoder into account)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	400 and returns the resulting encoded object. If this is the last call to
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	401 \method{encode} \var{final} must be true (the default is false).
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	402 \end{methoddesc}
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	403
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	404 \begin{methoddesc}{reset}{}
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	405 Reset the encoder to the initial state.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	406 \end{methoddesc}
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	407
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	408
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	409 \subsubsection{IncrementalDecoder Objects \label{incremental-decoder-objects}}
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	410
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	411 The \class{IncrementalDecoder} class is used for decoding an input in multiple
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	412 steps. It defines the following methods which every incremental decoder must
37888 5bc915feed14 Typo, grammar fixes. This file could use another proofreading pass. Andrew M. Kuchling <amk@amk.ca> parents: 37884 diff changeset	413 define in order to be compatible with the Python codec registry.
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	414
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	415 \begin{classdesc}{IncrementalDecoder}{\optional{errors}}
37902 52551221a132 Correct the grammar George Yoshida <dynkin@gmail.com> parents: 37899 diff changeset	416 Constructor for an \class{IncrementalDecoder} instance.
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	417
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	418 All incremental decoders must provide this constructor interface. They are
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	419 free to add additional keyword arguments, but only the ones defined
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	420 here are used by the Python codec registry.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	421
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	422 The \class{IncrementalDecoder} may implement different error handling
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	423 schemes by providing the \var{errors} keyword argument. These
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	424 parameters are predefined:
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	425
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	426 \begin{itemize}
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	427 \item \code{'strict'} Raise \exception{ValueError} (or a subclass);
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	428 this is the default.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	429 \item \code{'ignore'} Ignore the character and continue with the next.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	430 \item \code{'replace'} Replace with a suitable replacement character.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	431 \end{itemize}
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	432
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	433 The \var{errors} argument will be assigned to an attribute of the
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	434 same name. Assigning to this attribute makes it possible to switch
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	435 between different error handling strategies during the lifetime
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	436 of the \class{IncrementalEncoder} object.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	437
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	438 The set of allowed values for the \var{errors} argument can
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	439 be extended with \function{register_error()}.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	440 \end{classdesc}
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	441
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	442 \begin{methoddesc}{decode}{object\optional{, final}}
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	443 Decodes \var{object} (taking the current state of the decoder into account)
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	444 and returns the resulting decoded object. If this is the last call to
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	445 \method{decode} \var{final} must be true (the default is false).
37322 32e767699001 Clarify what the final argument does in Walter Dörwald <walter@livinglogic.de> parents: 37064 diff changeset	446 If \var{final} is true the decoder must decode the input completely and must
32e767699001 Clarify what the final argument does in Walter Dörwald <walter@livinglogic.de> parents: 37064 diff changeset	447 flush all buffers. If this isn't possible (e.g. because of incomplete byte
32e767699001 Clarify what the final argument does in Walter Dörwald <walter@livinglogic.de> parents: 37064 diff changeset	448 sequences at the end of the input) it must initiate error handling just like
32e767699001 Clarify what the final argument does in Walter Dörwald <walter@livinglogic.de> parents: 37064 diff changeset	449 in the stateless case (which might raise an exception).
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	450 \end{methoddesc}
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	451
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	452 \begin{methoddesc}{reset}{}
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	453 Reset the decoder to the initial state.
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	454 \end{methoddesc}
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	455
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36913 diff changeset	456
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	457 The \class{StreamWriter} and \class{StreamReader} classes provide
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	458 generic working interfaces which can be used to implement new
37888 5bc915feed14 Typo, grammar fixes. This file could use another proofreading pass. Andrew M. Kuchling <amk@amk.ca> parents: 37884 diff changeset	459 encoding submodules very easily. See \module{encodings.utf_8} for an
5bc915feed14 Typo, grammar fixes. This file could use another proofreading pass. Andrew M. Kuchling <amk@amk.ca> parents: 37884 diff changeset	460 example of how this is done.
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	461
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	462
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	463 \subsubsection{StreamWriter Objects \label{stream-writer-objects}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	464
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	465 The \class{StreamWriter} class is a subclass of \class{Codec} and
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	466 defines the following methods which every stream writer must define in
37888 5bc915feed14 Typo, grammar fixes. This file could use another proofreading pass. Andrew M. Kuchling <amk@amk.ca> parents: 37884 diff changeset	467 order to be compatible with the Python codec registry.
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	468
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	469 \begin{classdesc}{StreamWriter}{stream\optional{, errors}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	470 Constructor for a \class{StreamWriter} instance.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	471
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	472 All stream writers must provide this constructor interface. They are
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	473 free to add additional keyword arguments, but only the ones defined
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	474 here are used by the Python codec registry.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	475
37888 5bc915feed14 Typo, grammar fixes. This file could use another proofreading pass. Andrew M. Kuchling <amk@amk.ca> parents: 37884 diff changeset	476 \var{stream} must be a file-like object open for writing binary
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	477 data.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	478
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	479 The \class{StreamWriter} may implement different error handling
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	480 schemes by providing the \var{errors} keyword argument. These
26099 135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	481 parameters are predefined:
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	482
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	483 \begin{itemize}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	484 \item \code{'strict'} Raise \exception{ValueError} (or a subclass);
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	485 this is the default.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	486 \item \code{'ignore'} Ignore the character and continue with the next.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	487 \item \code{'replace'} Replace with a suitable replacement character
26099 135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	488 \item \code{'xmlcharrefreplace'} Replace with the appropriate XML
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	489 character reference
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	490 \item \code{'backslashreplace'} Replace with backslashed escape sequences.
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	491 \end{itemize}
26099 135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	492
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	493 The \var{errors} argument will be assigned to an attribute of the
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	494 same name. Assigning to this attribute makes it possible to switch
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	495 between different error handling strategies during the lifetime
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	496 of the \class{StreamWriter} object.
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	497
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	498 The set of allowed values for the \var{errors} argument can
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	499 be extended with \function{register_error()}.
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	500 \end{classdesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	501
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	502 \begin{methoddesc}{write}{object}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	503 Writes the object's contents encoded to the stream.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	504 \end{methoddesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	505
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	506 \begin{methoddesc}{writelines}{list}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	507 Writes the concatenated list of strings to the stream (possibly by
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	508 reusing the \method{write()} method).
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	509 \end{methoddesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	510
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	511 \begin{methoddesc}{reset}{}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	512 Flushes and resets the codec buffers used for keeping state.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	513
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	514 Calling this method should ensure that the data on the output is put
37888 5bc915feed14 Typo, grammar fixes. This file could use another proofreading pass. Andrew M. Kuchling <amk@amk.ca> parents: 37884 diff changeset	515 into a clean state that allows appending of new fresh data without
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	516 having to rescan the whole stream to recover state.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	517 \end{methoddesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	518
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	519 In addition to the above methods, the \class{StreamWriter} must also
37888 5bc915feed14 Typo, grammar fixes. This file could use another proofreading pass. Andrew M. Kuchling <amk@amk.ca> parents: 37884 diff changeset	520 inherit all other methods and attributes from the underlying stream.
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	521
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	522
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	523 \subsubsection{StreamReader Objects \label{stream-reader-objects}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	524
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	525 The \class{StreamReader} class is a subclass of \class{Codec} and
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	526 defines the following methods which every stream reader must define in
37888 5bc915feed14 Typo, grammar fixes. This file could use another proofreading pass. Andrew M. Kuchling <amk@amk.ca> parents: 37884 diff changeset	527 order to be compatible with the Python codec registry.
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	528
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	529 \begin{classdesc}{StreamReader}{stream\optional{, errors}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	530 Constructor for a \class{StreamReader} instance.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	531
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	532 All stream readers must provide this constructor interface. They are
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	533 free to add additional keyword arguments, but only the ones defined
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	534 here are used by the Python codec registry.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	535
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	536 \var{stream} must be a file-like object open for reading (binary)
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	537 data.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	538
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	539 The \class{StreamReader} may implement different error handling
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	540 schemes by providing the \var{errors} keyword argument. These
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	541 parameters are defined:
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	542
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	543 \begin{itemize}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	544 \item \code{'strict'} Raise \exception{ValueError} (or a subclass);
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	545 this is the default.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	546 \item \code{'ignore'} Ignore the character and continue with the next.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	547 \item \code{'replace'} Replace with a suitable replacement character.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	548 \end{itemize}
26099 135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	549
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	550 The \var{errors} argument will be assigned to an attribute of the
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	551 same name. Assigning to this attribute makes it possible to switch
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	552 between different error handling strategies during the lifetime
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	553 of the \class{StreamReader} object.
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	554
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	555 The set of allowed values for the \var{errors} argument can
135f08b73bd5 Add documentation for the PEP 293 functionality: Walter Dörwald <walter@livinglogic.de> parents: 25989 diff changeset	556 be extended with \function{register_error()}.
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	557 \end{classdesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	558
35416 4896f8334bff Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 34296 diff changeset	559 \begin{methoddesc}{read}{\optional{size\optional{, chars, \optional{firstline}}}}
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	560 Decodes data from the stream and returns the resulting object.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	561
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32727 diff changeset	562 \var{chars} indicates the number of characters to read from the
33437 fac2a53906f1 fix typo in markup Fred Drake <fdrake@acm.org> parents: 33428 diff changeset	563 stream. \function{read()} will never return more than \var{chars}
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32727 diff changeset	564 characters, but it might return less, if there are not enough
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32727 diff changeset	565 characters available.
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32727 diff changeset	566
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	567 \var{size} indicates the approximate maximum number of bytes to read
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	568 from the stream for decoding purposes. The decoder can modify this
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	569 setting as appropriate. The default value -1 indicates to read and
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	570 decode as much as possible. \var{size} is intended to prevent having
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	571 to decode huge files in one step.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	572
35416 4896f8334bff Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 34296 diff changeset	573 \var{firstline} indicates that it would be sufficient to only return
4896f8334bff Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 34296 diff changeset	574 the first line, if there are decoding errors on later lines.
4896f8334bff Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 34296 diff changeset	575
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	576 The method should use a greedy read strategy meaning that it should
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	577 read as much data as is allowed within the definition of the encoding
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	578 and the given size, e.g. if optional encoding endings or state
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	579 markers are available on the stream, these should be read too.
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32727 diff changeset	580
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32727 diff changeset	581 \versionchanged[\var{chars} argument added]{2.4}
35416 4896f8334bff Return complete lines from codec stream readers Martin v. Löwis <martin@v.loewis.de> parents: 34296 diff changeset	582 \versionchanged[\var{firstline} argument added]{2.4.2}
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	583 \end{methoddesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	584
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32727 diff changeset	585 \begin{methoddesc}{readline}{\optional{size\optional{, keepends}}}
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	586 Read one line from the input stream and return the
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	587 decoded data.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	588
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	589 \var{size}, if given, is passed as size argument to the stream's
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	590 \method{readline()} method.
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32727 diff changeset	591
37888 5bc915feed14 Typo, grammar fixes. This file could use another proofreading pass. Andrew M. Kuchling <amk@amk.ca> parents: 37884 diff changeset	592 If \var{keepends} is false line-endings will be stripped from the
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32727 diff changeset	593 lines returned.
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32727 diff changeset	594
57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32727 diff changeset	595 \versionchanged[\var{keepends} argument added]{2.4}
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	596 \end{methoddesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	597
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32727 diff changeset	598 \begin{methoddesc}{readlines}{\optional{sizehint\optional{, keepends}}}
37888 5bc915feed14 Typo, grammar fixes. This file could use another proofreading pass. Andrew M. Kuchling <amk@amk.ca> parents: 37884 diff changeset	599 Read all lines available on the input stream and return them as a list
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	600 of lines.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	601
37888 5bc915feed14 Typo, grammar fixes. This file could use another proofreading pass. Andrew M. Kuchling <amk@amk.ca> parents: 37884 diff changeset	602 Line-endings are implemented using the codec's decoder method and are
33428 57478a13fbf6 SF patch #998993: The UTF-8 and the UTF-16 stateful decoders now support Walter Dörwald <walter@livinglogic.de> parents: 32727 diff changeset	603 included in the list entries if \var{keepends} is true.
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	604
37888 5bc915feed14 Typo, grammar fixes. This file could use another proofreading pass. Andrew M. Kuchling <amk@amk.ca> parents: 37884 diff changeset	605 \var{sizehint}, if given, is passed as the \var{size} argument to the
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	606 stream's \method{read()} method.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	607 \end{methoddesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	608
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	609 \begin{methoddesc}{reset}{}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	610 Resets the codec buffers used for keeping state.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	611
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	612 Note that no stream repositioning should take place. This method is
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	613 primarily intended to be able to recover from decoding errors.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	614 \end{methoddesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	615
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	616 In addition to the above methods, the \class{StreamReader} must also
37888 5bc915feed14 Typo, grammar fixes. This file could use another proofreading pass. Andrew M. Kuchling <amk@amk.ca> parents: 37884 diff changeset	617 inherit all other methods and attributes from the underlying stream.
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	618
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	619 The next two base classes are included for convenience. They are not
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	620 needed by the codec registry, but may provide useful in practice.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	621
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	622
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	623 \subsubsection{StreamReaderWriter Objects \label{stream-reader-writer}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	624
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	625 The \class{StreamReaderWriter} allows wrapping streams which work in
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	626 both read and write modes.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	627
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	628 The design is such that one can use the factory functions returned by
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	629 the \function{lookup()} function to construct the instance.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	630
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	631 \begin{classdesc}{StreamReaderWriter}{stream, Reader, Writer, errors}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	632 Creates a \class{StreamReaderWriter} instance.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	633 \var{stream} must be a file-like object.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	634 \var{Reader} and \var{Writer} must be factory functions or classes
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	635 providing the \class{StreamReader} and \class{StreamWriter} interface
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	636 resp.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	637 Error handling is done in the same way as defined for the
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	638 stream readers and writers.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	639 \end{classdesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	640
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	641 \class{StreamReaderWriter} instances define the combined interfaces of
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	642 \class{StreamReader} and \class{StreamWriter} classes. They inherit
37888 5bc915feed14 Typo, grammar fixes. This file could use another proofreading pass. Andrew M. Kuchling <amk@amk.ca> parents: 37884 diff changeset	643 all other methods and attributes from the underlying stream.
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	644
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	645
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	646 \subsubsection{StreamRecoder Objects \label{stream-recoder-objects}}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	647
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	648 The \class{StreamRecoder} provide a frontend - backend view of
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	649 encoding data which is sometimes useful when dealing with different
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	650 encoding environments.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	651
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	652 The design is such that one can use the factory functions returned by
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	653 the \function{lookup()} function to construct the instance.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	654
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	655 \begin{classdesc}{StreamRecoder}{stream, encode, decode,
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	656 Reader, Writer, errors}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	657 Creates a \class{StreamRecoder} instance which implements a two-way
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	658 conversion: \var{encode} and \var{decode} work on the frontend (the
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	659 input to \method{read()} and output of \method{write()}) while
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	660 \var{Reader} and \var{Writer} work on the backend (reading and
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	661 writing to the stream).
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	662
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	663 You can use these objects to do transparent direct recodings from
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	664 e.g.\ Latin-1 to UTF-8 and back.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	665
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	666 \var{stream} must be a file-like object.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	667
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	668 \var{encode}, \var{decode} must adhere to the \class{Codec}
37888 5bc915feed14 Typo, grammar fixes. This file could use another proofreading pass. Andrew M. Kuchling <amk@amk.ca> parents: 37884 diff changeset	669 interface. \var{Reader}, \var{Writer} must be factory functions or
30134 523c3d61dbdd SF patch#786531 'the the' typo. Contributed by George Yoshida Raymond Hettinger <python@rcn.com> parents: 29894 diff changeset	670 classes providing objects of the \class{StreamReader} and
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	671 \class{StreamWriter} interface respectively.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	672
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	673 \var{encode} and \var{decode} are needed for the frontend
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	674 translation, \var{Reader} and \var{Writer} for the backend
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	675 translation. The intermediate format used is determined by the two
37888 5bc915feed14 Typo, grammar fixes. This file could use another proofreading pass. Andrew M. Kuchling <amk@amk.ca> parents: 37884 diff changeset	676 sets of codecs, e.g. the Unicode codecs will use Unicode as the
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	677 intermediate encoding.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	678
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	679 Error handling is done in the same way as defined for the
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	680 stream readers and writers.
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	681 \end{classdesc}
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	682
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	683 \class{StreamRecoder} instances define the combined interfaces of
b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	684 \class{StreamReader} and \class{StreamWriter} classes. They inherit
37888 5bc915feed14 Typo, grammar fixes. This file could use another proofreading pass. Andrew M. Kuchling <amk@amk.ca> parents: 37884 diff changeset	685 all other methods and attributes from the underlying stream.
15186 b2892cbc63db Marc-Andre Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 13696 diff changeset	686
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	687 \subsection{Encodings and Unicode\label{encodings-overview}}
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	688
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	689 Unicode strings are stored internally as sequences of codepoints (to
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	690 be precise as \ctype{Py_UNICODE} arrays). Depending on the way Python is
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	691 compiled (either via \longprogramopt{enable-unicode=ucs2} or
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	692 \longprogramopt{enable-unicode=ucs4}, with the former being the default)
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	693 \ctype{Py_UNICODE} is either a 16-bit or
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	694 32-bit data type. Once a Unicode object is used outside of CPU and
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	695 memory, CPU endianness and how these arrays are stored as bytes become
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	696 an issue. Transforming a unicode object into a sequence of bytes is
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	697 called encoding and recreating the unicode object from the sequence of
37888 5bc915feed14 Typo, grammar fixes. This file could use another proofreading pass. Andrew M. Kuchling <amk@amk.ca> parents: 37884 diff changeset	698 bytes is known as decoding. There are many different methods for how this
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	699 transformation can be done (these methods are also called encodings).
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	700 The simplest method is to map the codepoints 0-255 to the bytes
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	701 \code{0x0}-\code{0xff}. This means that a unicode object that contains
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	702 codepoints above \code{U+00FF} can't be encoded with this method (which
37064 64c83e0f683e More \exception fixes. Georg Brandl <georg@python.org> parents: 37024 diff changeset	703 is called \code{'latin-1'} or \code{'iso-8859-1'}).
64c83e0f683e More \exception fixes. Georg Brandl <georg@python.org> parents: 37024 diff changeset	704 \function{unicode.encode()} will raise a \exception{UnicodeEncodeError}
64c83e0f683e More \exception fixes. Georg Brandl <georg@python.org> parents: 37024 diff changeset	705 that looks like this: \samp{UnicodeEncodeError: 'latin-1' codec can't
64c83e0f683e More \exception fixes. Georg Brandl <georg@python.org> parents: 37024 diff changeset	706 encode character u'\e u1234' in position 3: ordinal not in range(256)}.
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	707
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	708 There's another group of encodings (the so called charmap encodings)
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	709 that choose a different subset of all unicode code points and how
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	710 these codepoints are mapped to the bytes \code{0x0}-\code{0xff.}
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	711 To see how this is done simply open e.g. \file{encodings/cp1252.py}
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	712 (which is an encoding that is used primarily on Windows).
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	713 There's a string constant with 256 characters that shows you which
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	714 character is mapped to which byte value.
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	715
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	716 All of these encodings can only encode 256 of the 65536 (or 1114111)
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	717 codepoints defined in unicode. A simple and straightforward way that
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	718 can store each Unicode code point, is to store each codepoint as two
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	719 consecutive bytes. There are two possibilities: Store the bytes in big
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	720 endian or in little endian order. These two encodings are called
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	721 UTF-16-BE and UTF-16-LE respectively. Their disadvantage is that if
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	722 e.g. you use UTF-16-BE on a little endian machine you will always have
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	723 to swap bytes on encoding and decoding. UTF-16 avoids this problem:
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	724 Bytes will always be in natural endianness. When these bytes are read
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	725 by a CPU with a different endianness, then bytes have to be swapped
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	726 though. To be able to detect the endianness of a UTF-16 byte sequence,
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	727 there's the so called BOM (the "Byte Order Mark"). This is the Unicode
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	728 character \code{U+FEFF}. This character will be prepended to every UTF-16
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	729 byte sequence. The byte swapped version of this character (\code{0xFFFE}) is
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	730 an illegal character that may not appear in a Unicode text. So when
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	731 the first character in an UTF-16 byte sequence appears to be a \code{U+FFFE}
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	732 the bytes have to be swapped on decoding. Unfortunately upto Unicode
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	733 4.0 the character \code{U+FEFF} had a second purpose as a \samp{ZERO WIDTH
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	734 NO-BREAK SPACE}: A character that has no width and doesn't allow a
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	735 word to be split. It can e.g. be used to give hints to a ligature
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	736 algorithm. With Unicode 4.0 using \code{U+FEFF} as a \samp{ZERO WIDTH NO-BREAK
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	737 SPACE} has been deprecated (with \code{U+2060} (\samp{WORD JOINER}) assuming
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	738 this role). Nevertheless Unicode software still must be able to handle
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	739 \code{U+FEFF} in both roles: As a BOM it's a device to determine the storage
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	740 layout of the encoded bytes, and vanishes once the byte sequence has
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	741 been decoded into a Unicode string; as a \samp{ZERO WIDTH NO-BREAK SPACE}
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	742 it's a normal character that will be decoded like any other.
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	743
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	744 There's another encoding that is able to encoding the full range of
37888 5bc915feed14 Typo, grammar fixes. This file could use another proofreading pass. Andrew M. Kuchling <amk@amk.ca> parents: 37884 diff changeset	745 Unicode characters: UTF-8. UTF-8 is an 8-bit encoding, which means
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	746 there are no issues with byte order in UTF-8. Each byte in a UTF-8
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	747 byte sequence consists of two parts: Marker bits (the most significant
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	748 bits) and payload bits. The marker bits are a sequence of zero to six
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	749 1 bits followed by a 0 bit. Unicode characters are encoded like this
36256 6cf7d04dcc1f Fix typos. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	750 (with x being payload bits, which when concatenated give the Unicode
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	751 character):
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	752
36637 d0ebcb4f6031 Fix table header. Walter Dörwald <walter@livinglogic.de> parents: 36389 diff changeset	753 \begin{tableii}{l\|l}{textrm}{Range}{Encoding}
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	754 \lineii{\code{U-00000000} ... \code{U-0000007F}}{0xxxxxxx}
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	755 \lineii{\code{U-00000080} ... \code{U-000007FF}}{110xxxxx 10xxxxxx}
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	756 \lineii{\code{U-00000800} ... \code{U-0000FFFF}}{1110xxxx 10xxxxxx 10xxxxxx}
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	757 \lineii{\code{U-00010000} ... \code{U-001FFFFF}}{11110xxx 10xxxxxx 10xxxxxx 10xxxxxx}
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	758 \lineii{\code{U-00200000} ... \code{U-03FFFFFF}}{111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx}
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	759 \lineii{\code{U-04000000} ... \code{U-7FFFFFFF}}{1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx}
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	760 \end{tableii}
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	761
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	762 The least significant bit of the Unicode character is the rightmost x
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	763 bit.
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	764
37888 5bc915feed14 Typo, grammar fixes. This file could use another proofreading pass. Andrew M. Kuchling <amk@amk.ca> parents: 37884 diff changeset	765 As UTF-8 is an 8-bit encoding no BOM is required and any \code{U+FEFF}
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	766 character in the decoded Unicode string (even if it's the first
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	767 character) is treated as a \samp{ZERO WIDTH NO-BREAK SPACE}.
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	768
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	769 Without external information it's impossible to reliably determine
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	770 which encoding was used for encoding a Unicode string. Each charmap
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	771 encoding can decode any random byte sequence. However that's not
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	772 possible with UTF-8, as UTF-8 byte sequences have a structure that
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	773 doesn't allow arbitrary byte sequence. To increase the reliability
36256 6cf7d04dcc1f Fix typos. Walter Dörwald <walter@livinglogic.de> parents: 36247 diff changeset	774 with which a UTF-8 encoding can be detected, Microsoft invented a
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	775 variant of UTF-8 (that Python 2.5 calls \code{"utf-8-sig"}) for its Notepad
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	776 program: Before any of the Unicode characters is written to the file,
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	777 a UTF-8 encoded BOM (which looks like this as a byte sequence: \code{0xef},
37888 5bc915feed14 Typo, grammar fixes. This file could use another proofreading pass. Andrew M. Kuchling <amk@amk.ca> parents: 37884 diff changeset	778 \code{0xbb}, \code{0xbf}) is written. As it's rather improbable that any
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	779 charmap encoded file starts with these byte values (which would e.g. map to
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	780
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	781 LATIN SMALL LETTER I WITH DIAERESIS \\
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	782 RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK \\
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	783 INVERTED QUESTION MARK
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	784
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	785 in iso-8859-1), this increases the probability that a utf-8-sig
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	786 encoding can be correctly guessed from the byte sequence. So here the
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	787 BOM is not used to be able to determine the byte order used for
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	788 generating the byte sequence, but as a signature that helps in
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	789 guessing the encoding. On encoding the utf-8-sig codec will write
36389 7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	790 \code{0xef}, \code{0xbb}, \code{0xbf} as the first three bytes to the file.
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	791 On decoding utf-8-sig will skip those three bytes if they appear as the
7f05974c8b80 Add markup to new section in codecs docs Georg Brandl <georg@python.org> parents: 36256 diff changeset	792 first three bytes in the file.
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	793
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	794
32458 b73ab309ada7 link to the codecs page from the "".encode() description. Skip Montanaro <skip@pobox.com> parents: 31687 diff changeset	795 \subsection{Standard Encodings\label{standard-encodings}}
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	796
37888 5bc915feed14 Typo, grammar fixes. This file could use another proofreading pass. Andrew M. Kuchling <amk@amk.ca> parents: 37884 diff changeset	797 Python comes with a number of codecs built-in, either implemented as C
5bc915feed14 Typo, grammar fixes. This file could use another proofreading pass. Andrew M. Kuchling <amk@amk.ca> parents: 37884 diff changeset	798 functions or with dictionaries as mapping tables. The following table
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	799 lists the codecs by name, together with a few common aliases, and the
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	800 languages for which the encoding is likely used. Neither the list of
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	801 aliases nor the list of languages is meant to be exhaustive. Notice
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	802 that spelling alternatives that only differ in case or use a hyphen
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	803 instead of an underscore are also valid aliases.
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	804
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	805 Many of the character sets support the same languages. They vary in
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	806 individual characters (e.g. whether the EURO SIGN is supported or
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	807 not), and in the assignment of characters to code positions. For the
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	808 European languages in particular, the following variants typically
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	809 exist:
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	810
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	811 \begin{itemize}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	812 \item an ISO 8859 codeset
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	813 \item a Microsoft Windows code page, which is typically derived from
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	814 a 8859 codeset, but replaces control characters with additional
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	815 graphic characters
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	816 \item an IBM EBCDIC code page
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	817 \item an IBM PC code page, which is \ASCII{} compatible
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	818 \end{itemize}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	819
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	820 \begin{longtableiii}{l\|l\|l}{textrm}{Codec}{Aliases}{Languages}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	821
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	822 \lineiii{ascii}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	823 {646, us-ascii}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	824 {English}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	825
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	826 \lineiii{big5}
32722 4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	827 {big5-tw, csbig5}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	828 {Traditional Chinese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	829
32727 69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	830 \lineiii{big5hkscs}
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	831 {big5-hkscs, hkscs}
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	832 {Traditional Chinese}
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	833
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	834 \lineiii{cp037}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	835 {IBM037, IBM039}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	836 {English}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	837
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	838 \lineiii{cp424}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	839 {EBCDIC-CP-HE, IBM424}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	840 {Hebrew}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	841
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	842 \lineiii{cp437}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	843 {437, IBM437}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	844 {English}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	845
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	846 \lineiii{cp500}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	847 {EBCDIC-CP-BE, EBCDIC-CP-CH, IBM500}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	848 {Western Europe}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	849
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	850 \lineiii{cp737}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	851 {}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	852 {Greek}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	853
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	854 \lineiii{cp775}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	855 {IBM775}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	856 {Baltic languages}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	857
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	858 \lineiii{cp850}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	859 {850, IBM850}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	860 {Western Europe}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	861
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	862 \lineiii{cp852}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	863 {852, IBM852}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	864 {Central and Eastern Europe}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	865
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	866 \lineiii{cp855}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	867 {855, IBM855}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	868 {Bulgarian, Byelorussian, Macedonian, Russian, Serbian}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	869
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	870 \lineiii{cp856}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	871 {}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	872 {Hebrew}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	873
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	874 \lineiii{cp857}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	875 {857, IBM857}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	876 {Turkish}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	877
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	878 \lineiii{cp860}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	879 {860, IBM860}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	880 {Portuguese}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	881
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	882 \lineiii{cp861}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	883 {861, CP-IS, IBM861}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	884 {Icelandic}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	885
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	886 \lineiii{cp862}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	887 {862, IBM862}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	888 {Hebrew}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	889
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	890 \lineiii{cp863}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	891 {863, IBM863}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	892 {Canadian}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	893
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	894 \lineiii{cp864}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	895 {IBM864}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	896 {Arabic}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	897
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	898 \lineiii{cp865}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	899 {865, IBM865}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	900 {Danish, Norwegian}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	901
32462 1b9da5fe9053 add cp866 row Skip Montanaro <skip@pobox.com> parents: 32458 diff changeset	902 \lineiii{cp866}
1b9da5fe9053 add cp866 row Skip Montanaro <skip@pobox.com> parents: 32458 diff changeset	903 {866, IBM866}
1b9da5fe9053 add cp866 row Skip Montanaro <skip@pobox.com> parents: 32458 diff changeset	904 {Russian}
1b9da5fe9053 add cp866 row Skip Montanaro <skip@pobox.com> parents: 32458 diff changeset	905
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	906 \lineiii{cp869}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	907 {869, CP-GR, IBM869}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	908 {Greek}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	909
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	910 \lineiii{cp874}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	911 {}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	912 {Thai}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	913
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	914 \lineiii{cp875}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	915 {}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	916 {Greek}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	917
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	918 \lineiii{cp932}
32722 4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	919 {932, ms932, mskanji, ms-kanji}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	920 {Japanese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	921
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	922 \lineiii{cp949}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	923 {949, ms949, uhc}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	924 {Korean}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	925
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	926 \lineiii{cp950}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	927 {950, ms950}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	928 {Traditional Chinese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	929
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	930 \lineiii{cp1006}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	931 {}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	932 {Urdu}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	933
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	934 \lineiii{cp1026}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	935 {ibm1026}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	936 {Turkish}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	937
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	938 \lineiii{cp1140}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	939 {ibm1140}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	940 {Western Europe}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	941
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	942 \lineiii{cp1250}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	943 {windows-1250}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	944 {Central and Eastern Europe}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	945
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	946 \lineiii{cp1251}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	947 {windows-1251}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	948 {Bulgarian, Byelorussian, Macedonian, Russian, Serbian}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	949
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	950 \lineiii{cp1252}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	951 {windows-1252}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	952 {Western Europe}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	953
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	954 \lineiii{cp1253}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	955 {windows-1253}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	956 {Greek}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	957
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	958 \lineiii{cp1254}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	959 {windows-1254}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	960 {Turkish}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	961
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	962 \lineiii{cp1255}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	963 {windows-1255}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	964 {Hebrew}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	965
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	966 \lineiii{cp1256}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	967 {windows1256}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	968 {Arabic}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	969
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	970 \lineiii{cp1257}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	971 {windows-1257}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	972 {Baltic languages}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	973
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	974 \lineiii{cp1258}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	975 {windows-1258}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	976 {Vietnamese}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	977
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	978 \lineiii{euc_jp}
32722 4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	979 {eucjp, ujis, u-jis}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	980 {Japanese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	981
32727 69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	982 \lineiii{euc_jis_2004}
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	983 {jisx0213, eucjis2004}
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	984 {Japanese}
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	985
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	986 \lineiii{euc_jisx0213}
32727 69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	987 {eucjisx0213}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	988 {Japanese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	989
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	990 \lineiii{euc_kr}
32722 4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	991 {euckr, korean, ksc5601, ks_c-5601, ks_c-5601-1987, ksx1001, ks_x-1001}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	992 {Korean}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	993
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	994 \lineiii{gb2312}
32722 4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	995 {chinese, csiso58gb231280, euc-cn, euccn, eucgb2312-cn, gb2312-1980,
4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	996 gb2312-80, iso-ir-58}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	997 {Simplified Chinese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	998
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	999 \lineiii{gbk}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1000 {936, cp936, ms936}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1001 {Unified Chinese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1002
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1003 \lineiii{gb18030}
32722 4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	1004 {gb18030-2000}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1005 {Unified Chinese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1006
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1007 \lineiii{hz}
32722 4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	1008 {hzgb, hz-gb, hz-gb-2312}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1009 {Simplified Chinese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1010
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1011 \lineiii{iso2022_jp}
32722 4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	1012 {csiso2022jp, iso2022jp, iso-2022-jp}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1013 {Japanese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1014
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1015 \lineiii{iso2022_jp_1}
32722 4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	1016 {iso2022jp-1, iso-2022-jp-1}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1017 {Japanese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1018
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1019 \lineiii{iso2022_jp_2}
32722 4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	1020 {iso2022jp-2, iso-2022-jp-2}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1021 {Japanese, Korean, Simplified Chinese, Western Europe, Greek}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1022
32727 69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	1023 \lineiii{iso2022_jp_2004}
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	1024 {iso2022jp-2004, iso-2022-jp-2004}
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	1025 {Japanese}
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	1026
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1027 \lineiii{iso2022_jp_3}
32722 4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	1028 {iso2022jp-3, iso-2022-jp-3}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1029 {Japanese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1030
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1031 \lineiii{iso2022_jp_ext}
32722 4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	1032 {iso2022jp-ext, iso-2022-jp-ext}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1033 {Japanese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1034
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1035 \lineiii{iso2022_kr}
32722 4d43b0abc22a Change CJK encoding aliases to their most popular variation of Hye-Shik Chang <hyeshik@gmail.com> parents: 32462 diff changeset	1036 {csiso2022kr, iso2022kr, iso-2022-kr}
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1037 {Korean}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1038
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1039 \lineiii{latin_1}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1040 {iso-8859-1, iso8859-1, 8859, cp819, latin, latin1, L1}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1041 {West Europe}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1042
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1043 \lineiii{iso8859_2}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1044 {iso-8859-2, latin2, L2}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1045 {Central and Eastern Europe}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1046
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1047 \lineiii{iso8859_3}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1048 {iso-8859-3, latin3, L3}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1049 {Esperanto, Maltese}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1050
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1051 \lineiii{iso8859_4}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1052 {iso-8859-4, latin4, L4}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1053 {Baltic languagues}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1054
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1055 \lineiii{iso8859_5}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1056 {iso-8859-5, cyrillic}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1057 {Bulgarian, Byelorussian, Macedonian, Russian, Serbian}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1058
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1059 \lineiii{iso8859_6}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1060 {iso-8859-6, arabic}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1061 {Arabic}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1062
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1063 \lineiii{iso8859_7}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1064 {iso-8859-7, greek, greek8}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1065 {Greek}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1066
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1067 \lineiii{iso8859_8}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1068 {iso-8859-8, hebrew}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1069 {Hebrew}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1070
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1071 \lineiii{iso8859_9}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1072 {iso-8859-9, latin5, L5}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1073 {Turkish}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1074
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1075 \lineiii{iso8859_10}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1076 {iso-8859-10, latin6, L6}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1077 {Nordic languages}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1078
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1079 \lineiii{iso8859_13}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1080 {iso-8859-13}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1081 {Baltic languages}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1082
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1083 \lineiii{iso8859_14}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1084 {iso-8859-14, latin8, L8}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1085 {Celtic languages}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1086
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1087 \lineiii{iso8859_15}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1088 {iso-8859-15}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1089 {Western Europe}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1090
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1091 \lineiii{johab}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1092 {cp1361, ms1361}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1093 {Korean}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1094
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1095 \lineiii{koi8_r}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1096 {}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1097 {Russian}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1098
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1099 \lineiii{koi8_u}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1100 {}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1101 {Ukrainian}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1102
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1103 \lineiii{mac_cyrillic}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1104 {maccyrillic}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1105 {Bulgarian, Byelorussian, Macedonian, Russian, Serbian}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1106
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1107 \lineiii{mac_greek}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1108 {macgreek}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1109 {Greek}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1110
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1111 \lineiii{mac_iceland}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1112 {maciceland}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1113 {Icelandic}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1114
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1115 \lineiii{mac_latin2}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1116 {maclatin2, maccentraleurope}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1117 {Central and Eastern Europe}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1118
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1119 \lineiii{mac_roman}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1120 {macroman}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1121 {Western Europe}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1122
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1123 \lineiii{mac_turkish}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1124 {macturkish}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1125 {Turkish}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1126
31687 a885e4d64ad0 Add a new unicode codec: ptcp154 (Kazakh) Hye-Shik Chang <hyeshik@gmail.com> parents: 31386 diff changeset	1127 \lineiii{ptcp154}
a885e4d64ad0 Add a new unicode codec: ptcp154 (Kazakh) Hye-Shik Chang <hyeshik@gmail.com> parents: 31386 diff changeset	1128 {csptcp154, pt154, cp154, cyrillic-asian}
a885e4d64ad0 Add a new unicode codec: ptcp154 (Kazakh) Hye-Shik Chang <hyeshik@gmail.com> parents: 31386 diff changeset	1129 {Kazakh}
a885e4d64ad0 Add a new unicode codec: ptcp154 (Kazakh) Hye-Shik Chang <hyeshik@gmail.com> parents: 31386 diff changeset	1130
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1131 \lineiii{shift_jis}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1132 {csshiftjis, shiftjis, sjis, s_jis}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1133 {Japanese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1134
32727 69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	1135 \lineiii{shift_jis_2004}
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	1136 {shiftjis2004, sjis_2004, sjis2004}
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	1137 {Japanese}
69dadc2ca14d Bring CJKCodecs 1.1 into trunk. This completely reorganizes source Hye-Shik Chang <hyeshik@gmail.com> parents: 32722 diff changeset	1138
31386 887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1139 \lineiii{shift_jisx0213}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1140 {shiftjisx0213, sjisx0213, s_jisx0213}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1141 {Japanese}
887ce39f95f2 Add CJK codecs support as discussed on python-dev. (SF #873597) Hye-Shik Chang <hyeshik@gmail.com> parents: 30522 diff changeset	1142
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1143 \lineiii{utf_16}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1144 {U16, utf16}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1145 {all languages}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1146
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1147 \lineiii{utf_16_be}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1148 {UTF-16BE}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1149 {all languages (BMP only)}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1150
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1151 \lineiii{utf_16_le}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1152 {UTF-16LE}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1153 {all languages (BMP only)}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1154
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1155 \lineiii{utf_7}
35710 59c4a65642c5 Bug #1245379: Add "unicode-1-1-utf-7" as an alias for "utf-7" as specified Walter Dörwald <walter@livinglogic.de> parents: 35416 diff changeset	1156 {U7, unicode-1-1-utf-7}
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1157 {all languages}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1158
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1159 \lineiii{utf_8}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1160 {U8, UTF, utf8}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1161 {all languages}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1162
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1163 \lineiii{utf_8_sig}
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1164 {}
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1165 {all languages}
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1166
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1167 \end{longtableiii}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1168
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1169 A number of codecs are specific to Python, so their codec names have
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1170 no meaning outside Python. Some of them don't convert from Unicode
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1171 strings to byte strings, but instead use the property of the Python
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1172 codecs machinery that any bijective function with one argument can be
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1173 considered as an encoding.
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1174
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1175 For the codecs listed below, the result in the ``encoding'' direction
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1176 is always a byte string. The result of the ``decoding'' direction is
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1177 listed as operand type in the table.
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1178
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1179 \begin{tableiv}{l\|l\|l\|l}{textrm}{Codec}{Aliases}{Operand type}{Purpose}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1180
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1181 \lineiv{base64_codec}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1182 {base64, base-64}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1183 {byte string}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1184 {Convert operand to MIME base64}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1185
30522 db63cb0fc7da Added codec for bz2 compression. Raymond Hettinger <python@rcn.com> parents: 30505 diff changeset	1186 \lineiv{bz2_codec}
db63cb0fc7da Added codec for bz2 compression. Raymond Hettinger <python@rcn.com> parents: 30505 diff changeset	1187 {bz2}
db63cb0fc7da Added codec for bz2 compression. Raymond Hettinger <python@rcn.com> parents: 30505 diff changeset	1188 {byte string}
db63cb0fc7da Added codec for bz2 compression. Raymond Hettinger <python@rcn.com> parents: 30505 diff changeset	1189 {Compress the operand using bz2}
db63cb0fc7da Added codec for bz2 compression. Raymond Hettinger <python@rcn.com> parents: 30505 diff changeset	1190
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1191 \lineiv{hex_codec}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1192 {hex}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1193 {byte string}
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1194 {Convert operand to hexadecimal representation, with two
1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1195 digits per byte}
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1196
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1197 \lineiv{idna}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1198 {}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1199 {Unicode string}
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1200 {Implements \rfc{3490}.
30273 244b55c90e0f Minor typo Raymond Hettinger <python@rcn.com> parents: 30134 diff changeset	1201 \versionadded{2.3}
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1202 See also \refmodule{encodings.idna}}
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1203
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1204 \lineiv{mbcs}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1205 {dbcs}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1206 {Unicode string}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1207 {Windows only: Encode operand according to the ANSI codepage (CP_ACP)}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1208
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1209 \lineiv{palmos}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1210 {}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1211 {Unicode string}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1212 {Encoding of PalmOS 3.5}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1213
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1214 \lineiv{punycode}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1215 {}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1216 {Unicode string}
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1217 {Implements \rfc{3492}.
1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1218 \versionadded{2.3}}
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1219
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1220 \lineiv{quopri_codec}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1221 {quopri, quoted-printable, quotedprintable}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1222 {byte string}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1223 {Convert operand to MIME quoted printable}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1224
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1225 \lineiv{raw_unicode_escape}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1226 {}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1227 {Unicode string}
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1228 {Produce a string that is suitable as raw Unicode literal in
1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1229 Python source code}
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1230
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1231 \lineiv{rot_13}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1232 {rot13}
37522 5556ce99f20c Bug #1465619: rot13 codec decodes to a Unicode string. Georg Brandl <georg@python.org> parents: 37322 diff changeset	1233 {Unicode string}
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1234 {Returns the Caesar-cypher encryption of the operand}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1235
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1236 \lineiv{string_escape}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1237 {}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1238 {byte string}
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1239 {Produce a string that is suitable as string literal in
1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1240 Python source code}
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1241
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1242 \lineiv{undefined}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1243 {}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1244 {any}
36913 0e36e7c66b8e typo Georg Brandl <georg@python.org> parents: 36637 diff changeset	1245 {Raise an exception for all conversions. Can be used as the
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1246 system encoding if no automatic coercion between byte and
1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1247 Unicode strings is desired.}
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1248
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1249 \lineiv{unicode_escape}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1250 {}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1251 {Unicode string}
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1252 {Produce a string that is suitable as Unicode literal in
1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1253 Python source code}
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1254
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1255 \lineiv{unicode_internal}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1256 {}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1257 {Unicode string}
34296 67f2190e1f6d SF Patch #1093896: miscellaneous doc typos Raymond Hettinger <python@rcn.com> parents: 33437 diff changeset	1258 {Return the internal representation of the operand}
26943 55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1259
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1260 \lineiv{uu_codec}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1261 {uu}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1262 {byte string}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1263 {Convert the operand using uuencode}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1264
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1265 \lineiv{zlib_codec}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1266 {zip, zlib}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1267 {byte string}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1268 {Compress the operand using gzip}
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1269
55141144dab5 Document standard encodings. Martin v. Löwis <martin@v.loewis.de> parents: 26294 diff changeset	1270 \end{tableiv}
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1271
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1272 \subsection{\module{encodings.idna} ---
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1273 Internationalized Domain Names in Applications}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1274
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1275 \declaremodule{standard}{encodings.idna}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1276 \modulesynopsis{Internationalized Domain Names implementation}
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1277 % XXX The next line triggers a formatting bug, so it's commented out
1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1278 % until that can be fixed.
1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1279 %\moduleauthor{Martin v. L\"owis}
1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1280
1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1281 \versionadded{2.3}
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1282
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1283 This module implements \rfc{3490} (Internationalized Domain Names in
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1284 Applications) and \rfc{3492} (Nameprep: A Stringprep Profile for
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1285 Internationalized Domain Names (IDN)). It builds upon the
29894 ac84f179bb84 A variety of markup-level adjustments. Fred Drake <fdrake@acm.org> parents: 29528 diff changeset	1286 \code{punycode} encoding and \refmodule{stringprep}.
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1287
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1288 These RFCs together define a protocol to support non-\ASCII{} characters
1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1289 in domain names. A domain name containing non-\ASCII{} characters (such
29894 ac84f179bb84 A variety of markup-level adjustments. Fred Drake <fdrake@acm.org> parents: 29528 diff changeset	1290 as ``www.Alliancefran\c caise.nu'') is converted into an
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1291 \ASCII-compatible encoding (ACE, such as
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1292 ``www.xn--alliancefranaise-npb.nu''). The ACE form of the domain name
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1293 is then used in all places where arbitrary characters are not allowed
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1294 by the protocol, such as DNS queries, HTTP \mailheader{Host} fields, and so
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1295 on. This conversion is carried out in the application; if possible
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1296 invisible to the user: The application should transparently convert
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1297 Unicode domain labels to IDNA on the wire, and convert back ACE labels
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1298 to Unicode before presenting them to the user.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1299
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1300 Python supports this conversion in several ways: The \code{idna} codec
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1301 allows to convert between Unicode and the ACE. Furthermore, the
29894 ac84f179bb84 A variety of markup-level adjustments. Fred Drake <fdrake@acm.org> parents: 29528 diff changeset	1302 \refmodule{socket} module transparently converts Unicode host names to
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1303 ACE, so that applications need not be concerned about converting host
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1304 names themselves when they pass them to the socket module. On top of
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1305 that, modules that have host names as function parameters, such as
29894 ac84f179bb84 A variety of markup-level adjustments. Fred Drake <fdrake@acm.org> parents: 29528 diff changeset	1306 \refmodule{httplib} and \refmodule{ftplib}, accept Unicode host names
ac84f179bb84 A variety of markup-level adjustments. Fred Drake <fdrake@acm.org> parents: 29528 diff changeset	1307 (\refmodule{httplib} then also transparently sends an IDNA hostname in
ac84f179bb84 A variety of markup-level adjustments. Fred Drake <fdrake@acm.org> parents: 29528 diff changeset	1308 the \mailheader{Host} field if it sends that field at all).
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1309
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1310 When receiving host names from the wire (such as in reverse name
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1311 lookup), no automatic conversion to Unicode is performed: Applications
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1312 wishing to present such host names to the user should decode them to
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1313 Unicode.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1314
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1315 The module \module{encodings.idna} also implements the nameprep
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1316 procedure, which performs certain normalizations on host names, to
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1317 achieve case-insensitivity of international domain names, and to unify
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1318 similar characters. The nameprep functions can be used directly if
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1319 desired.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1320
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1321 \begin{funcdesc}{nameprep}{label}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1322 Return the nameprepped version of \var{label}. The implementation
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1323 currently assumes query strings, so \code{AllowUnassigned} is
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1324 true.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1325 \end{funcdesc}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1326
29528 7aee6ef596dd Fix typo. Raymond Hettinger <python@rcn.com> parents: 28906 diff changeset	1327 \begin{funcdesc}{ToASCII}{label}
28906 1eac1a35b370 - comment out \moduleauthor that broke formatting until the formatting Fred Drake <fdrake@acm.org> parents: 28682 diff changeset	1328 Convert a label to \ASCII, as specified in \rfc{3490}.
28671 d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1329 \code{UseSTD3ASCIIRules} is assumed to be false.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1330 \end{funcdesc}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1331
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1332 \begin{funcdesc}{ToUnicode}{label}
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1333 Convert a label to Unicode, as specified in \rfc{3490}.
d41d8de9768f Implement IDNA (Internationalized Domain Names in Applications). Martin v. Löwis <martin@v.loewis.de> parents: 27555 diff changeset	1334 \end{funcdesc}
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1335
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1336 \subsection{\module{encodings.utf_8_sig} ---
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1337 UTF-8 codec with BOM signature}
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1338 \declaremodule{standard}{encodings.utf-8-sig} % XXX utf_8_sig gives TeX errors
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1339 \modulesynopsis{UTF-8 codec with BOM signature}
37899 37c28a6ad501 Fix a markup. George Yoshida <dynkin@gmail.com> parents: 37888 diff changeset	1340 \moduleauthor{Walter D\"orwald}{}
36247 3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1341
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1342 \versionadded{2.5}
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1343
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1344 This module implements a variant of the UTF-8 codec: On encoding a
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1345 UTF-8 encoded BOM will be prepended to the UTF-8 encoded bytes. For
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1346 the stateful encoder this is only done once (on the first write to the
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1347 byte stream). For decoding an optional UTF-8 encoded BOM at the start
3ba9ff79bb29 Patch #1177307: UTF-8-Sig codec. Martin v. Löwis <martin@v.loewis.de> parents: 35951 diff changeset	1348 of the data will be skipped.

Mercurial > cpython-lru_cache-weakref

annotate Doc/lib/libcodecs.tex @ 39042:c02fa187e0f4 legacy-trunk