cpython-lru_cache-weakref: Doc/library/codecs.rst annotate

annotate Doc/library/codecs.rst @ 54294:dba767bc3155 legacy-trunk

remove mention of old ctypes version

author	Benjamin Peterson <benjamin@python.org>
date	Sat, 23 May 2009 20:59:09 +0000
parents	82ccc7e77ec5
children	85aa706f7126 0ed405c05cc3

rev	line source
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	2 :mod:`codecs` --- Codec registry and base classes
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	3 =================================================
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	4
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	5 .. module:: codecs
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	6 :synopsis: Encode and decode data and streams.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	7 .. moduleauthor:: Marc-Andre Lemburg <mal@lemburg.com>
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	8 .. sectionauthor:: Marc-Andre Lemburg <mal@lemburg.com>
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	9 .. sectionauthor:: Martin v. Löwis <martin@v.loewis.de>
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	10
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	11
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	12 .. index::
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	13 single: Unicode
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	14 single: Codecs
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	15 pair: Codecs; encode
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	16 pair: Codecs; decode
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	17 single: streams
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	18 pair: stackable; streams
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	19
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	20 This module defines base classes for standard Python codecs (encoders and
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	21 decoders) and provides access to the internal Python codec registry which
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	22 manages the codec and error handling lookup process.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	23
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	24 It defines the following functions:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	25
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	26
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	27 .. function:: register(search_function)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	28
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	29 Register a codec search function. Search functions are expected to take one
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	30 argument, the encoding name in all lower case letters, and return a
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	31 :class:`CodecInfo` object having the following attributes:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	32
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	33 * ``name`` The name of the encoding;
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	34
49677 82ccc7e77ec5 Merged revisions 66801,66803-66804,66813,66854-66856,66866,66870-66872,66874,66887,66903,66905,66911,66913,66927,66932,66938,66942,66962,66964,66973-66974,66977,66992,66998-66999,67002,67005,67007,67028,67040-67041,67044,67070,67089,67091,67101,67117-67119,67123-67124 via svnmerge from Georg Brandl <georg@python.org> parents: 48919 diff changeset	35 * ``encode`` The stateless encoding function;
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	36
49677 82ccc7e77ec5 Merged revisions 66801,66803-66804,66813,66854-66856,66866,66870-66872,66874,66887,66903,66905,66911,66913,66927,66932,66938,66942,66962,66964,66973-66974,66977,66992,66998-66999,67002,67005,67007,67028,67040-67041,67044,67070,67089,67091,67101,67117-67119,67123-67124 via svnmerge from Georg Brandl <georg@python.org> parents: 48919 diff changeset	37 * ``decode`` The stateless decoding function;
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	38
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	39 * ``incrementalencoder`` An incremental encoder class or factory function;
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	40
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	41 * ``incrementaldecoder`` An incremental decoder class or factory function;
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	42
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	43 * ``streamwriter`` A stream writer class or factory function;
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	44
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	45 * ``streamreader`` A stream reader class or factory function.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	46
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	47 The various functions or classes take the following arguments:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	48
49677 82ccc7e77ec5 Merged revisions 66801,66803-66804,66813,66854-66856,66866,66870-66872,66874,66887,66903,66905,66911,66913,66927,66932,66938,66942,66962,66964,66973-66974,66977,66992,66998-66999,67002,67005,67007,67028,67040-67041,67044,67070,67089,67091,67101,67117-67119,67123-67124 via svnmerge from Georg Brandl <georg@python.org> parents: 48919 diff changeset	49 encode and decode: These must be functions or methods which have the same
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	50 interface as the :meth:`encode`/:meth:`decode` methods of Codec instances (see
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	51 Codec Interface). The functions/methods are expected to work in a stateless
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	52 mode.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	53
48919 13fdf98ea290 #3749: fix c'n'p errors. Georg Brandl <georg@python.org> parents: 47340 diff changeset	54 incrementalencoder and incrementaldecoder: These have to be factory
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	55 functions providing the following interface:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	56
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	57 ``factory(errors='strict')``
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	58
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	59 The factory functions must return objects providing the interfaces defined by
48919 13fdf98ea290 #3749: fix c'n'p errors. Georg Brandl <georg@python.org> parents: 47340 diff changeset	60 the base classes :class:`IncrementalEncoder` and :class:`IncrementalDecoder`,
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	61 respectively. Incremental codecs can maintain state.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	62
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	63 streamreader and streamwriter: These have to be factory functions providing
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	64 the following interface:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	65
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	66 ``factory(stream, errors='strict')``
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	67
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	68 The factory functions must return objects providing the interfaces defined by
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	69 the base classes :class:`StreamWriter` and :class:`StreamReader`, respectively.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	70 Stream codecs can maintain state.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	71
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	72 Possible values for errors are ``'strict'`` (raise an exception in case of an
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	73 encoding error), ``'replace'`` (replace malformed data with a suitable
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	74 replacement marker, such as ``'?'``), ``'ignore'`` (ignore malformed data and
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	75 continue without further notice), ``'xmlcharrefreplace'`` (replace with the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	76 appropriate XML character reference (for encoding only)) and
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	77 ``'backslashreplace'`` (replace with backslashed escape sequences (for encoding
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	78 only)) as well as any other error handling name defined via
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	79 :func:`register_error`.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	80
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	81 In case a search function cannot find a given encoding, it should return
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	82 ``None``.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	83
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	84
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	85 .. function:: lookup(encoding)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	86
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	87 Looks up the codec info in the Python codec registry and returns a
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	88 :class:`CodecInfo` object as defined above.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	89
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	90 Encodings are first looked up in the registry's cache. If not found, the list of
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	91 registered search functions is scanned. If no :class:`CodecInfo` object is
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	92 found, a :exc:`LookupError` is raised. Otherwise, the :class:`CodecInfo` object
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	93 is stored in the cache and returned to the caller.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	94
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	95 To simplify access to the various codecs, the module provides these additional
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	96 functions which use :func:`lookup` for the codec lookup:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	97
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	98
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	99 .. function:: getencoder(encoding)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	100
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	101 Look up the codec for the given encoding and return its encoder function.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	102
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	103 Raises a :exc:`LookupError` in case the encoding cannot be found.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	104
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	105
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	106 .. function:: getdecoder(encoding)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	107
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	108 Look up the codec for the given encoding and return its decoder function.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	109
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	110 Raises a :exc:`LookupError` in case the encoding cannot be found.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	111
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	112
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	113 .. function:: getincrementalencoder(encoding)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	114
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	115 Look up the codec for the given encoding and return its incremental encoder
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	116 class or factory function.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	117
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	118 Raises a :exc:`LookupError` in case the encoding cannot be found or the codec
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	119 doesn't support an incremental encoder.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	120
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	121 .. versionadded:: 2.5
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	122
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	123
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	124 .. function:: getincrementaldecoder(encoding)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	125
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	126 Look up the codec for the given encoding and return its incremental decoder
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	127 class or factory function.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	128
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	129 Raises a :exc:`LookupError` in case the encoding cannot be found or the codec
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	130 doesn't support an incremental decoder.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	131
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	132 .. versionadded:: 2.5
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	133
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	134
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	135 .. function:: getreader(encoding)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	136
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	137 Look up the codec for the given encoding and return its StreamReader class or
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	138 factory function.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	139
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	140 Raises a :exc:`LookupError` in case the encoding cannot be found.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	141
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	142
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	143 .. function:: getwriter(encoding)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	144
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	145 Look up the codec for the given encoding and return its StreamWriter class or
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	146 factory function.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	147
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	148 Raises a :exc:`LookupError` in case the encoding cannot be found.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	149
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	150
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	151 .. function:: register_error(name, error_handler)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	152
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	153 Register the error handling function error_handler under the name name.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	154 error_handler will be called during encoding and decoding in case of an error,
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	155 when name is specified as the errors parameter.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	156
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	157 For encoding error_handler will be called with a :exc:`UnicodeEncodeError`
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	158 instance, which contains information about the location of the error. The error
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	159 handler must either raise this or a different exception or return a tuple with a
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	160 replacement for the unencodable part of the input and a position where encoding
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	161 should continue. The encoder will encode the replacement and continue encoding
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	162 the original input at the specified position. Negative position values will be
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	163 treated as being relative to the end of the input string. If the resulting
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	164 position is out of bound an :exc:`IndexError` will be raised.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	165
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	166 Decoding and translating works similar, except :exc:`UnicodeDecodeError` or
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	167 :exc:`UnicodeTranslateError` will be passed to the handler and that the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	168 replacement from the error handler will be put into the output directly.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	169
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	170
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	171 .. function:: lookup_error(name)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	172
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	173 Return the error handler previously registered under the name name.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	174
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	175 Raises a :exc:`LookupError` in case the handler cannot be found.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	176
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	177
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	178 .. function:: strict_errors(exception)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	179
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	180 Implements the ``strict`` error handling.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	181
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	182
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	183 .. function:: replace_errors(exception)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	184
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	185 Implements the ``replace`` error handling.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	186
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	187
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	188 .. function:: ignore_errors(exception)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	189
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	190 Implements the ``ignore`` error handling.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	191
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	192
43184 ad3d5acacf03 Fix wrong function names. Walter Dörwald <walter@livinglogic.de> parents: 42649 diff changeset	193 .. function:: xmlcharrefreplace_errors(exception)
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	194
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	195 Implements the ``xmlcharrefreplace`` error handling.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	196
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	197
43184 ad3d5acacf03 Fix wrong function names. Walter Dörwald <walter@livinglogic.de> parents: 42649 diff changeset	198 .. function:: backslashreplace_errors(exception)
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	199
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	200 Implements the ``backslashreplace`` error handling.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	201
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	202 To simplify working with encoded files or stream, the module also defines these
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	203 utility functions:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	204
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	205
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	206 .. function:: open(filename, mode[, encoding[, errors[, buffering]]])
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	207
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	208 Open an encoded file using the given mode and return a wrapped version
45376 6212367390e3 #2131: note that codecs.open() always opens files in binary mode. Georg Brandl <georg@python.org> parents: 44103 diff changeset	209 providing transparent encoding/decoding. The default file mode is ``'r'``
6212367390e3 #2131: note that codecs.open() always opens files in binary mode. Georg Brandl <georg@python.org> parents: 44103 diff changeset	210 meaning to open the file in read mode.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	211
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	212 .. note::
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	213
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	214 The wrapped version will only accept the object format defined by the codecs,
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	215 i.e. Unicode objects for most built-in codecs. Output is also codec-dependent
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	216 and will usually be Unicode as well.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	217
45376 6212367390e3 #2131: note that codecs.open() always opens files in binary mode. Georg Brandl <georg@python.org> parents: 44103 diff changeset	218 .. note::
6212367390e3 #2131: note that codecs.open() always opens files in binary mode. Georg Brandl <georg@python.org> parents: 44103 diff changeset	219
6212367390e3 #2131: note that codecs.open() always opens files in binary mode. Georg Brandl <georg@python.org> parents: 44103 diff changeset	220 Files are always opened in binary mode, even if no binary mode was
6212367390e3 #2131: note that codecs.open() always opens files in binary mode. Georg Brandl <georg@python.org> parents: 44103 diff changeset	221 specified. This is done to avoid data loss due to encodings using 8-bit
6212367390e3 #2131: note that codecs.open() always opens files in binary mode. Georg Brandl <georg@python.org> parents: 44103 diff changeset	222 values. This means that no automatic conversion of ``'\n'`` is done
6212367390e3 #2131: note that codecs.open() always opens files in binary mode. Georg Brandl <georg@python.org> parents: 44103 diff changeset	223 on reading and writing.
6212367390e3 #2131: note that codecs.open() always opens files in binary mode. Georg Brandl <georg@python.org> parents: 44103 diff changeset	224
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	225 encoding specifies the encoding which is to be used for the file.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	226
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	227 errors may be given to define the error handling. It defaults to ``'strict'``
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	228 which causes a :exc:`ValueError` to be raised in case an encoding error occurs.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	229
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	230 buffering has the same meaning as for the built-in :func:`open` function. It
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	231 defaults to line buffered.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	232
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	233
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	234 .. function:: EncodedFile(file, input[, output[, errors]])
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	235
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	236 Return a wrapped version of file which provides transparent encoding
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	237 translation.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	238
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	239 Strings written to the wrapped file are interpreted according to the given
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	240 input encoding and then written to the original file as strings using the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	241 output encoding. The intermediate encoding will usually be Unicode but depends
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	242 on the specified codecs.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	243
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	244 If output is not given, it defaults to input.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	245
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	246 errors may be given to define the error handling. It defaults to ``'strict'``,
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	247 which causes :exc:`ValueError` to be raised in case an encoding error occurs.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	248
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	249
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	250 .. function:: iterencode(iterable, encoding[, errors])
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	251
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	252 Uses an incremental encoder to iteratively encode the input provided by
43629 4f662ba09f1f Add :term: for generators. Georg Brandl <georg@python.org> parents: 43185 diff changeset	253 iterable. This function is a :term:`generator`. errors (as well as any
4f662ba09f1f Add :term: for generators. Georg Brandl <georg@python.org> parents: 43185 diff changeset	254 other keyword argument) is passed through to the incremental encoder.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	255
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	256 .. versionadded:: 2.5
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	257
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	258
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	259 .. function:: iterdecode(iterable, encoding[, errors])
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	260
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	261 Uses an incremental decoder to iteratively decode the input provided by
43629 4f662ba09f1f Add :term: for generators. Georg Brandl <georg@python.org> parents: 43185 diff changeset	262 iterable. This function is a :term:`generator`. errors (as well as any
4f662ba09f1f Add :term: for generators. Georg Brandl <georg@python.org> parents: 43185 diff changeset	263 other keyword argument) is passed through to the incremental decoder.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	264
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	265 .. versionadded:: 2.5
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	266
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	267 The module also provides the following constants which are useful for reading
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	268 and writing to platform dependent files:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	269
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	270
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	271 .. data:: BOM
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	272 BOM_BE
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	273 BOM_LE
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	274 BOM_UTF8
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	275 BOM_UTF16
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	276 BOM_UTF16_BE
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	277 BOM_UTF16_LE
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	278 BOM_UTF32
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	279 BOM_UTF32_BE
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	280 BOM_UTF32_LE
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	281
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	282 These constants define various encodings of the Unicode byte order mark (BOM)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	283 used in UTF-16 and UTF-32 data streams to indicate the byte order used in the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	284 stream or file and in UTF-8 as a Unicode signature. :const:`BOM_UTF16` is either
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	285 :const:`BOM_UTF16_BE` or :const:`BOM_UTF16_LE` depending on the platform's
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	286 native byte order, :const:`BOM` is an alias for :const:`BOM_UTF16`,
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	287 :const:`BOM_LE` for :const:`BOM_UTF16_LE` and :const:`BOM_BE` for
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	288 :const:`BOM_UTF16_BE`. The others represent the BOM in UTF-8 and UTF-32
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	289 encodings.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	290
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	291
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	292 .. _codec-base-classes:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	293
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	294 Codec Base Classes
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	295 ------------------
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	296
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	297 The :mod:`codecs` module defines a set of base classes which define the
47340 f6c88f172bbe fix minor grammar typo Benjamin Peterson <benjamin@python.org> parents: 46555 diff changeset	298 interface and can also be used to easily write your own codecs for use in
f6c88f172bbe fix minor grammar typo Benjamin Peterson <benjamin@python.org> parents: 46555 diff changeset	299 Python.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	300
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	301 Each codec has to define four interfaces to make it usable as codec in Python:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	302 stateless encoder, stateless decoder, stream reader and stream writer. The
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	303 stream reader and writers typically reuse the stateless encoder/decoder to
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	304 implement the file protocols.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	305
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	306 The :class:`Codec` class defines the interface for stateless encoders/decoders.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	307
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	308 To simplify and standardize error handling, the :meth:`encode` and
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	309 :meth:`decode` methods may implement different error handling schemes by
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	310 providing the errors string argument. The following string values are defined
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	311 and implemented by all standard Python codecs:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	312
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	313 +-------------------------+-----------------------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	314 \| Value \| Meaning \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	315 +=========================+===============================================+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	316 \| ``'strict'`` \| Raise :exc:`UnicodeError` (or a subclass); \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	317 \| \| this is the default. \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	318 +-------------------------+-----------------------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	319 \| ``'ignore'`` \| Ignore the character and continue with the \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	320 \| \| next. \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	321 +-------------------------+-----------------------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	322 \| ``'replace'`` \| Replace with a suitable replacement \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	323 \| \| character; Python will use the official \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	324 \| \| U+FFFD REPLACEMENT CHARACTER for the built-in \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	325 \| \| Unicode codecs on decoding and '?' on \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	326 \| \| encoding. \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	327 +-------------------------+-----------------------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	328 \| ``'xmlcharrefreplace'`` \| Replace with the appropriate XML character \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	329 \| \| reference (only for encoding). \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	330 +-------------------------+-----------------------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	331 \| ``'backslashreplace'`` \| Replace with backslashed escape sequences \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	332 \| \| (only for encoding). \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	333 +-------------------------+-----------------------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	334
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	335 The set of allowed values can be extended via :meth:`register_error`.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	336
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	337
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	338 .. _codec-objects:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	339
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	340 Codec Objects
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	341 ^^^^^^^^^^^^^
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	342
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	343 The :class:`Codec` class defines these methods which also define the function
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	344 interfaces of the stateless encoder and decoder:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	345
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	346
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	347 .. method:: Codec.encode(input[, errors])
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	348
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	349 Encodes the object input and returns a tuple (output object, length consumed).
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	350 While codecs are not restricted to use with Unicode, in a Unicode context,
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	351 encoding converts a Unicode object to a plain string using a particular
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	352 character set encoding (e.g., ``cp1252`` or ``iso-8859-1``).
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	353
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	354 errors defines the error handling to apply. It defaults to ``'strict'``
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	355 handling.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	356
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	357 The method may not store state in the :class:`Codec` instance. Use
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	358 :class:`StreamCodec` for codecs which have to keep state in order to make
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	359 encoding/decoding efficient.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	360
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	361 The encoder must be able to handle zero length input and return an empty object
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	362 of the output object type in this situation.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	363
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	364
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	365 .. method:: Codec.decode(input[, errors])
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	366
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	367 Decodes the object input and returns a tuple (output object, length consumed).
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	368 In a Unicode context, decoding converts a plain string encoded using a
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	369 particular character set encoding to a Unicode object.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	370
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	371 input must be an object which provides the ``bf_getreadbuf`` buffer slot.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	372 Python strings, buffer objects and memory mapped files are examples of objects
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	373 providing this slot.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	374
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	375 errors defines the error handling to apply. It defaults to ``'strict'``
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	376 handling.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	377
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	378 The method may not store state in the :class:`Codec` instance. Use
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	379 :class:`StreamCodec` for codecs which have to keep state in order to make
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	380 encoding/decoding efficient.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	381
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	382 The decoder must be able to handle zero length input and return an empty object
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	383 of the output object type in this situation.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	384
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	385 The :class:`IncrementalEncoder` and :class:`IncrementalDecoder` classes provide
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	386 the basic interface for incremental encoding and decoding. Encoding/decoding the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	387 input isn't done with one call to the stateless encoder/decoder function, but
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	388 with multiple calls to the :meth:`encode`/:meth:`decode` method of the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	389 incremental encoder/decoder. The incremental encoder/decoder keeps track of the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	390 encoding/decoding process during method calls.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	391
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	392 The joined output of calls to the :meth:`encode`/:meth:`decode` method is the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	393 same as if all the single inputs were joined into one, and this input was
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	394 encoded/decoded with the stateless encoder/decoder.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	395
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	396
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	397 .. _incremental-encoder-objects:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	398
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	399 IncrementalEncoder Objects
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	400 ^^^^^^^^^^^^^^^^^^^^^^^^^^
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	401
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	402 .. versionadded:: 2.5
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	403
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	404 The :class:`IncrementalEncoder` class is used for encoding an input in multiple
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	405 steps. It defines the following methods which every incremental encoder must
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	406 define in order to be compatible with the Python codec registry.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	407
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	408
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	409 .. class:: IncrementalEncoder([errors])
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	410
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	411 Constructor for an :class:`IncrementalEncoder` instance.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	412
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	413 All incremental encoders must provide this constructor interface. They are free
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	414 to add additional keyword arguments, but only the ones defined here are used by
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	415 the Python codec registry.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	416
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	417 The :class:`IncrementalEncoder` may implement different error handling schemes
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	418 by providing the errors keyword argument. These parameters are predefined:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	419
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	420 * ``'strict'`` Raise :exc:`ValueError` (or a subclass); this is the default.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	421
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	422 * ``'ignore'`` Ignore the character and continue with the next.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	423
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	424 * ``'replace'`` Replace with a suitable replacement character
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	425
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	426 * ``'xmlcharrefreplace'`` Replace with the appropriate XML character reference
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	427
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	428 * ``'backslashreplace'`` Replace with backslashed escape sequences.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	429
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	430 The errors argument will be assigned to an attribute of the same name.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	431 Assigning to this attribute makes it possible to switch between different error
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	432 handling strategies during the lifetime of the :class:`IncrementalEncoder`
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	433 object.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	434
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	435 The set of allowed values for the errors argument can be extended with
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	436 :func:`register_error`.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	437
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	438
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	439 .. method:: encode(object[, final])
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	440
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	441 Encodes object (taking the current state of the encoder into account)
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	442 and returns the resulting encoded object. If this is the last call to
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	443 :meth:`encode` final must be true (the default is false).
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	444
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	445
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	446 .. method:: reset()
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	447
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	448 Reset the encoder to the initial state.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	449
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	450
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	451 .. _incremental-decoder-objects:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	452
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	453 IncrementalDecoder Objects
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	454 ^^^^^^^^^^^^^^^^^^^^^^^^^^
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	455
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	456 The :class:`IncrementalDecoder` class is used for decoding an input in multiple
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	457 steps. It defines the following methods which every incremental decoder must
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	458 define in order to be compatible with the Python codec registry.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	459
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	460
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	461 .. class:: IncrementalDecoder([errors])
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	462
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	463 Constructor for an :class:`IncrementalDecoder` instance.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	464
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	465 All incremental decoders must provide this constructor interface. They are free
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	466 to add additional keyword arguments, but only the ones defined here are used by
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	467 the Python codec registry.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	468
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	469 The :class:`IncrementalDecoder` may implement different error handling schemes
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	470 by providing the errors keyword argument. These parameters are predefined:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	471
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	472 * ``'strict'`` Raise :exc:`ValueError` (or a subclass); this is the default.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	473
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	474 * ``'ignore'`` Ignore the character and continue with the next.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	475
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	476 * ``'replace'`` Replace with a suitable replacement character.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	477
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	478 The errors argument will be assigned to an attribute of the same name.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	479 Assigning to this attribute makes it possible to switch between different error
48919 13fdf98ea290 #3749: fix c'n'p errors. Georg Brandl <georg@python.org> parents: 47340 diff changeset	480 handling strategies during the lifetime of the :class:`IncrementalDecoder`
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	481 object.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	482
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	483 The set of allowed values for the errors argument can be extended with
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	484 :func:`register_error`.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	485
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	486
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	487 .. method:: decode(object[, final])
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	488
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	489 Decodes object (taking the current state of the decoder into account)
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	490 and returns the resulting decoded object. If this is the last call to
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	491 :meth:`decode` final must be true (the default is false). If final is
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	492 true the decoder must decode the input completely and must flush all
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	493 buffers. If this isn't possible (e.g. because of incomplete byte sequences
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	494 at the end of the input) it must initiate error handling just like in the
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	495 stateless case (which might raise an exception).
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	496
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	497
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	498 .. method:: reset()
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	499
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	500 Reset the decoder to the initial state.
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	501
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	502
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	503 The :class:`StreamWriter` and :class:`StreamReader` classes provide generic
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	504 working interfaces which can be used to implement new encoding submodules very
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	505 easily. See :mod:`encodings.utf_8` for an example of how this is done.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	506
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	507
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	508 .. _stream-writer-objects:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	509
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	510 StreamWriter Objects
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	511 ^^^^^^^^^^^^^^^^^^^^
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	512
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	513 The :class:`StreamWriter` class is a subclass of :class:`Codec` and defines the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	514 following methods which every stream writer must define in order to be
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	515 compatible with the Python codec registry.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	516
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	517
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	518 .. class:: StreamWriter(stream[, errors])
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	519
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	520 Constructor for a :class:`StreamWriter` instance.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	521
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	522 All stream writers must provide this constructor interface. They are free to add
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	523 additional keyword arguments, but only the ones defined here are used by the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	524 Python codec registry.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	525
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	526 stream must be a file-like object open for writing binary data.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	527
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	528 The :class:`StreamWriter` may implement different error handling schemes by
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	529 providing the errors keyword argument. These parameters are predefined:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	530
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	531 * ``'strict'`` Raise :exc:`ValueError` (or a subclass); this is the default.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	532
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	533 * ``'ignore'`` Ignore the character and continue with the next.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	534
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	535 * ``'replace'`` Replace with a suitable replacement character
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	536
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	537 * ``'xmlcharrefreplace'`` Replace with the appropriate XML character reference
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	538
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	539 * ``'backslashreplace'`` Replace with backslashed escape sequences.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	540
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	541 The errors argument will be assigned to an attribute of the same name.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	542 Assigning to this attribute makes it possible to switch between different error
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	543 handling strategies during the lifetime of the :class:`StreamWriter` object.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	544
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	545 The set of allowed values for the errors argument can be extended with
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	546 :func:`register_error`.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	547
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	548
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	549 .. method:: write(object)
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	550
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	551 Writes the object's contents encoded to the stream.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	552
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	553
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	554 .. method:: writelines(list)
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	555
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	556 Writes the concatenated list of strings to the stream (possibly by reusing
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	557 the :meth:`write` method).
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	558
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	559
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	560 .. method:: reset()
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	561
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	562 Flushes and resets the codec buffers used for keeping state.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	563
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	564 Calling this method should ensure that the data on the output is put into
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	565 a clean state that allows appending of new fresh data without having to
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	566 rescan the whole stream to recover state.
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	567
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	568
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	569 In addition to the above methods, the :class:`StreamWriter` must also inherit
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	570 all other methods and attributes from the underlying stream.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	571
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	572
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	573 .. _stream-reader-objects:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	574
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	575 StreamReader Objects
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	576 ^^^^^^^^^^^^^^^^^^^^
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	577
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	578 The :class:`StreamReader` class is a subclass of :class:`Codec` and defines the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	579 following methods which every stream reader must define in order to be
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	580 compatible with the Python codec registry.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	581
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	582
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	583 .. class:: StreamReader(stream[, errors])
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	584
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	585 Constructor for a :class:`StreamReader` instance.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	586
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	587 All stream readers must provide this constructor interface. They are free to add
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	588 additional keyword arguments, but only the ones defined here are used by the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	589 Python codec registry.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	590
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	591 stream must be a file-like object open for reading (binary) data.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	592
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	593 The :class:`StreamReader` may implement different error handling schemes by
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	594 providing the errors keyword argument. These parameters are defined:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	595
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	596 * ``'strict'`` Raise :exc:`ValueError` (or a subclass); this is the default.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	597
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	598 * ``'ignore'`` Ignore the character and continue with the next.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	599
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	600 * ``'replace'`` Replace with a suitable replacement character.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	601
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	602 The errors argument will be assigned to an attribute of the same name.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	603 Assigning to this attribute makes it possible to switch between different error
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	604 handling strategies during the lifetime of the :class:`StreamReader` object.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	605
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	606 The set of allowed values for the errors argument can be extended with
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	607 :func:`register_error`.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	608
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	609
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	610 .. method:: read([size[, chars, [firstline]]])
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	611
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	612 Decodes data from the stream and returns the resulting object.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	613
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	614 chars indicates the number of characters to read from the
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	615 stream. :func:`read` will never return more than chars characters, but
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	616 it might return less, if there are not enough characters available.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	617
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	618 size indicates the approximate maximum number of bytes to read from the
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	619 stream for decoding purposes. The decoder can modify this setting as
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	620 appropriate. The default value -1 indicates to read and decode as much as
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	621 possible. size is intended to prevent having to decode huge files in
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	622 one step.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	623
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	624 firstline indicates that it would be sufficient to only return the first
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	625 line, if there are decoding errors on later lines.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	626
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	627 The method should use a greedy read strategy meaning that it should read
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	628 as much data as is allowed within the definition of the encoding and the
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	629 given size, e.g. if optional encoding endings or state markers are
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	630 available on the stream, these should be read too.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	631
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	632 .. versionchanged:: 2.4
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	633 chars argument added.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	634
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	635 .. versionchanged:: 2.4.2
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	636 firstline argument added.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	637
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	638
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	639 .. method:: readline([size[, keepends]])
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	640
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	641 Read one line from the input stream and return the decoded data.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	642
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	643 size, if given, is passed as size argument to the stream's
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	644 :meth:`readline` method.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	645
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	646 If keepends is false line-endings will be stripped from the lines
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	647 returned.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	648
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	649 .. versionchanged:: 2.4
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	650 keepends argument added.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	651
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	652
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	653 .. method:: readlines([sizehint[, keepends]])
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	654
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	655 Read all lines available on the input stream and return them as a list of
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	656 lines.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	657
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	658 Line-endings are implemented using the codec's decoder method and are
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	659 included in the list entries if keepends is true.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	660
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	661 sizehint, if given, is passed as the size argument to the stream's
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	662 :meth:`read` method.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	663
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	664
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	665 .. method:: reset()
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	666
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	667 Resets the codec buffers used for keeping state.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	668
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	669 Note that no stream repositioning should take place. This method is
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	670 primarily intended to be able to recover from decoding errors.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	671
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	672
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	673 In addition to the above methods, the :class:`StreamReader` must also inherit
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	674 all other methods and attributes from the underlying stream.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	675
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	676 The next two base classes are included for convenience. They are not needed by
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	677 the codec registry, but may provide useful in practice.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	678
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	679
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	680 .. _stream-reader-writer:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	681
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	682 StreamReaderWriter Objects
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	683 ^^^^^^^^^^^^^^^^^^^^^^^^^^
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	684
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	685 The :class:`StreamReaderWriter` allows wrapping streams which work in both read
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	686 and write modes.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	687
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	688 The design is such that one can use the factory functions returned by the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	689 :func:`lookup` function to construct the instance.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	690
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	691
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	692 .. class:: StreamReaderWriter(stream, Reader, Writer, errors)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	693
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	694 Creates a :class:`StreamReaderWriter` instance. stream must be a file-like
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	695 object. Reader and Writer must be factory functions or classes providing the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	696 :class:`StreamReader` and :class:`StreamWriter` interface resp. Error handling
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	697 is done in the same way as defined for the stream readers and writers.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	698
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	699 :class:`StreamReaderWriter` instances define the combined interfaces of
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	700 :class:`StreamReader` and :class:`StreamWriter` classes. They inherit all other
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	701 methods and attributes from the underlying stream.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	702
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	703
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	704 .. _stream-recoder-objects:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	705
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	706 StreamRecoder Objects
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	707 ^^^^^^^^^^^^^^^^^^^^^
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	708
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	709 The :class:`StreamRecoder` provide a frontend - backend view of encoding data
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	710 which is sometimes useful when dealing with different encoding environments.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	711
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	712 The design is such that one can use the factory functions returned by the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	713 :func:`lookup` function to construct the instance.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	714
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	715
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	716 .. class:: StreamRecoder(stream, encode, decode, Reader, Writer, errors)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	717
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	718 Creates a :class:`StreamRecoder` instance which implements a two-way conversion:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	719 encode and decode work on the frontend (the input to :meth:`read` and output
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	720 of :meth:`write`) while Reader and Writer work on the backend (reading and
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	721 writing to the stream).
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	722
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	723 You can use these objects to do transparent direct recodings from e.g. Latin-1
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	724 to UTF-8 and back.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	725
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	726 stream must be a file-like object.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	727
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	728 encode, decode must adhere to the :class:`Codec` interface. Reader,
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	729 Writer must be factory functions or classes providing objects of the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	730 :class:`StreamReader` and :class:`StreamWriter` interface respectively.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	731
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	732 encode and decode are needed for the frontend translation, Reader and
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	733 Writer for the backend translation. The intermediate format used is
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	734 determined by the two sets of codecs, e.g. the Unicode codecs will use Unicode
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	735 as the intermediate encoding.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	736
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	737 Error handling is done in the same way as defined for the stream readers and
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	738 writers.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	739
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	740
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	741 :class:`StreamRecoder` instances define the combined interfaces of
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	742 :class:`StreamReader` and :class:`StreamWriter` classes. They inherit all other
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	743 methods and attributes from the underlying stream.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	744
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	745
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	746 .. _encodings-overview:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	747
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	748 Encodings and Unicode
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	749 ---------------------
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	750
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	751 Unicode strings are stored internally as sequences of codepoints (to be precise
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	752 as :ctype:`Py_UNICODE` arrays). Depending on the way Python is compiled (either
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	753 via :option:`--enable-unicode=ucs2` or :option:`--enable-unicode=ucs4`, with the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	754 former being the default) :ctype:`Py_UNICODE` is either a 16-bit or 32-bit data
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	755 type. Once a Unicode object is used outside of CPU and memory, CPU endianness
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	756 and how these arrays are stored as bytes become an issue. Transforming a
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	757 unicode object into a sequence of bytes is called encoding and recreating the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	758 unicode object from the sequence of bytes is known as decoding. There are many
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	759 different methods for how this transformation can be done (these methods are
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	760 also called encodings). The simplest method is to map the codepoints 0-255 to
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	761 the bytes ``0x0``-``0xff``. This means that a unicode object that contains
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	762 codepoints above ``U+00FF`` can't be encoded with this method (which is called
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	763 ``'latin-1'`` or ``'iso-8859-1'``). :func:`unicode.encode` will raise a
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	764 :exc:`UnicodeEncodeError` that looks like this: ``UnicodeEncodeError: 'latin-1'
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	765 codec can't encode character u'\u1234' in position 3: ordinal not in
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	766 range(256)``.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	767
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	768 There's another group of encodings (the so called charmap encodings) that choose
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	769 a different subset of all unicode code points and how these codepoints are
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	770 mapped to the bytes ``0x0``-``0xff``. To see how this is done simply open
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	771 e.g. :file:`encodings/cp1252.py` (which is an encoding that is used primarily on
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	772 Windows). There's a string constant with 256 characters that shows you which
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	773 character is mapped to which byte value.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	774
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	775 All of these encodings can only encode 256 of the 65536 (or 1114111) codepoints
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	776 defined in unicode. A simple and straightforward way that can store each Unicode
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	777 code point, is to store each codepoint as two consecutive bytes. There are two
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	778 possibilities: Store the bytes in big endian or in little endian order. These
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	779 two encodings are called UTF-16-BE and UTF-16-LE respectively. Their
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	780 disadvantage is that if e.g. you use UTF-16-BE on a little endian machine you
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	781 will always have to swap bytes on encoding and decoding. UTF-16 avoids this
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	782 problem: Bytes will always be in natural endianness. When these bytes are read
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	783 by a CPU with a different endianness, then bytes have to be swapped though. To
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	784 be able to detect the endianness of a UTF-16 byte sequence, there's the so
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	785 called BOM (the "Byte Order Mark"). This is the Unicode character ``U+FEFF``.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	786 This character will be prepended to every UTF-16 byte sequence. The byte swapped
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	787 version of this character (``0xFFFE``) is an illegal character that may not
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	788 appear in a Unicode text. So when the first character in an UTF-16 byte sequence
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	789 appears to be a ``U+FFFE`` the bytes have to be swapped on decoding.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	790 Unfortunately upto Unicode 4.0 the character ``U+FEFF`` had a second purpose as
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	791 a ``ZERO WIDTH NO-BREAK SPACE``: A character that has no width and doesn't allow
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	792 a word to be split. It can e.g. be used to give hints to a ligature algorithm.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	793 With Unicode 4.0 using ``U+FEFF`` as a ``ZERO WIDTH NO-BREAK SPACE`` has been
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	794 deprecated (with ``U+2060`` (``WORD JOINER``) assuming this role). Nevertheless
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	795 Unicode software still must be able to handle ``U+FEFF`` in both roles: As a BOM
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	796 it's a device to determine the storage layout of the encoded bytes, and vanishes
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	797 once the byte sequence has been decoded into a Unicode string; as a ``ZERO WIDTH
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	798 NO-BREAK SPACE`` it's a normal character that will be decoded like any other.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	799
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	800 There's another encoding that is able to encoding the full range of Unicode
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	801 characters: UTF-8. UTF-8 is an 8-bit encoding, which means there are no issues
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	802 with byte order in UTF-8. Each byte in a UTF-8 byte sequence consists of two
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	803 parts: Marker bits (the most significant bits) and payload bits. The marker bits
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	804 are a sequence of zero to six 1 bits followed by a 0 bit. Unicode characters are
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	805 encoded like this (with x being payload bits, which when concatenated give the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	806 Unicode character):
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	807
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	808 +-----------------------------------+----------------------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	809 \| Range \| Encoding \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	810 +===================================+==============================================+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	811 \| ``U-00000000`` ... ``U-0000007F`` \| 0xxxxxxx \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	812 +-----------------------------------+----------------------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	813 \| ``U-00000080`` ... ``U-000007FF`` \| 110xxxxx 10xxxxxx \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	814 +-----------------------------------+----------------------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	815 \| ``U-00000800`` ... ``U-0000FFFF`` \| 1110xxxx 10xxxxxx 10xxxxxx \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	816 +-----------------------------------+----------------------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	817 \| ``U-00010000`` ... ``U-001FFFFF`` \| 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	818 +-----------------------------------+----------------------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	819 \| ``U-00200000`` ... ``U-03FFFFFF`` \| 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	820 +-----------------------------------+----------------------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	821 \| ``U-04000000`` ... ``U-7FFFFFFF`` \| 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	822 \| \| 10xxxxxx \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	823 +-----------------------------------+----------------------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	824
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	825 The least significant bit of the Unicode character is the rightmost x bit.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	826
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	827 As UTF-8 is an 8-bit encoding no BOM is required and any ``U+FEFF`` character in
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	828 the decoded Unicode string (even if it's the first character) is treated as a
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	829 ``ZERO WIDTH NO-BREAK SPACE``.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	830
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	831 Without external information it's impossible to reliably determine which
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	832 encoding was used for encoding a Unicode string. Each charmap encoding can
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	833 decode any random byte sequence. However that's not possible with UTF-8, as
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	834 UTF-8 byte sequences have a structure that doesn't allow arbitrary byte
43185 9b1833fc5c7b Fix typo. Walter Dörwald <walter@livinglogic.de> parents: 43184 diff changeset	835 sequences. To increase the reliability with which a UTF-8 encoding can be
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	836 detected, Microsoft invented a variant of UTF-8 (that Python 2.5 calls
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	837 ``"utf-8-sig"``) for its Notepad program: Before any of the Unicode characters
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	838 is written to the file, a UTF-8 encoded BOM (which looks like this as a byte
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	839 sequence: ``0xef``, ``0xbb``, ``0xbf``) is written. As it's rather improbable
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	840 that any charmap encoded file starts with these byte values (which would e.g.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	841 map to
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	842
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	843 \| LATIN SMALL LETTER I WITH DIAERESIS
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	844 \| RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	845 \| INVERTED QUESTION MARK
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	846
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	847 in iso-8859-1), this increases the probability that a utf-8-sig encoding can be
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	848 correctly guessed from the byte sequence. So here the BOM is not used to be able
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	849 to determine the byte order used for generating the byte sequence, but as a
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	850 signature that helps in guessing the encoding. On encoding the utf-8-sig codec
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	851 will write ``0xef``, ``0xbb``, ``0xbf`` as the first three bytes to the file. On
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	852 decoding utf-8-sig will skip those three bytes if they appear as the first three
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	853 bytes in the file.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	854
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	855
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	856 .. _standard-encodings:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	857
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	858 Standard Encodings
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	859 ------------------
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	860
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	861 Python comes with a number of codecs built-in, either implemented as C functions
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	862 or with dictionaries as mapping tables. The following table lists the codecs by
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	863 name, together with a few common aliases, and the languages for which the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	864 encoding is likely used. Neither the list of aliases nor the list of languages
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	865 is meant to be exhaustive. Notice that spelling alternatives that only differ in
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	866 case or use a hyphen instead of an underscore are also valid aliases.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	867
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	868 Many of the character sets support the same languages. They vary in individual
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	869 characters (e.g. whether the EURO SIGN is supported or not), and in the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	870 assignment of characters to code positions. For the European languages in
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	871 particular, the following variants typically exist:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	872
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	873 * an ISO 8859 codeset
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	874
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	875 * a Microsoft Windows code page, which is typically derived from a 8859 codeset,
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	876 but replaces control characters with additional graphic characters
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	877
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	878 * an IBM EBCDIC code page
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	879
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	880 * an IBM PC code page, which is ASCII compatible
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	881
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	882 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	883 \| Codec \| Aliases \| Languages \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	884 +=================+================================+================================+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	885 \| ascii \| 646, us-ascii \| English \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	886 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	887 \| big5 \| big5-tw, csbig5 \| Traditional Chinese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	888 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	889 \| big5hkscs \| big5-hkscs, hkscs \| Traditional Chinese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	890 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	891 \| cp037 \| IBM037, IBM039 \| English \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	892 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	893 \| cp424 \| EBCDIC-CP-HE, IBM424 \| Hebrew \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	894 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	895 \| cp437 \| 437, IBM437 \| English \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	896 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	897 \| cp500 \| EBCDIC-CP-BE, EBCDIC-CP-CH, \| Western Europe \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	898 \| \| IBM500 \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	899 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	900 \| cp737 \| \| Greek \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	901 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	902 \| cp775 \| IBM775 \| Baltic languages \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	903 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	904 \| cp850 \| 850, IBM850 \| Western Europe \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	905 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	906 \| cp852 \| 852, IBM852 \| Central and Eastern Europe \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	907 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	908 \| cp855 \| 855, IBM855 \| Bulgarian, Byelorussian, \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	909 \| \| \| Macedonian, Russian, Serbian \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	910 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	911 \| cp856 \| \| Hebrew \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	912 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	913 \| cp857 \| 857, IBM857 \| Turkish \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	914 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	915 \| cp860 \| 860, IBM860 \| Portuguese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	916 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	917 \| cp861 \| 861, CP-IS, IBM861 \| Icelandic \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	918 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	919 \| cp862 \| 862, IBM862 \| Hebrew \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	920 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	921 \| cp863 \| 863, IBM863 \| Canadian \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	922 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	923 \| cp864 \| IBM864 \| Arabic \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	924 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	925 \| cp865 \| 865, IBM865 \| Danish, Norwegian \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	926 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	927 \| cp866 \| 866, IBM866 \| Russian \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	928 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	929 \| cp869 \| 869, CP-GR, IBM869 \| Greek \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	930 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	931 \| cp874 \| \| Thai \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	932 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	933 \| cp875 \| \| Greek \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	934 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	935 \| cp932 \| 932, ms932, mskanji, ms-kanji \| Japanese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	936 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	937 \| cp949 \| 949, ms949, uhc \| Korean \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	938 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	939 \| cp950 \| 950, ms950 \| Traditional Chinese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	940 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	941 \| cp1006 \| \| Urdu \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	942 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	943 \| cp1026 \| ibm1026 \| Turkish \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	944 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	945 \| cp1140 \| ibm1140 \| Western Europe \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	946 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	947 \| cp1250 \| windows-1250 \| Central and Eastern Europe \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	948 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	949 \| cp1251 \| windows-1251 \| Bulgarian, Byelorussian, \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	950 \| \| \| Macedonian, Russian, Serbian \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	951 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	952 \| cp1252 \| windows-1252 \| Western Europe \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	953 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	954 \| cp1253 \| windows-1253 \| Greek \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	955 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	956 \| cp1254 \| windows-1254 \| Turkish \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	957 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	958 \| cp1255 \| windows-1255 \| Hebrew \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	959 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	960 \| cp1256 \| windows1256 \| Arabic \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	961 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	962 \| cp1257 \| windows-1257 \| Baltic languages \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	963 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	964 \| cp1258 \| windows-1258 \| Vietnamese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	965 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	966 \| euc_jp \| eucjp, ujis, u-jis \| Japanese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	967 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	968 \| euc_jis_2004 \| jisx0213, eucjis2004 \| Japanese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	969 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	970 \| euc_jisx0213 \| eucjisx0213 \| Japanese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	971 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	972 \| euc_kr \| euckr, korean, ksc5601, \| Korean \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	973 \| \| ks_c-5601, ks_c-5601-1987, \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	974 \| \| ksx1001, ks_x-1001 \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	975 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	976 \| gb2312 \| chinese, csiso58gb231280, euc- \| Simplified Chinese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	977 \| \| cn, euccn, eucgb2312-cn, \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	978 \| \| gb2312-1980, gb2312-80, iso- \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	979 \| \| ir-58 \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	980 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	981 \| gbk \| 936, cp936, ms936 \| Unified Chinese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	982 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	983 \| gb18030 \| gb18030-2000 \| Unified Chinese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	984 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	985 \| hz \| hzgb, hz-gb, hz-gb-2312 \| Simplified Chinese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	986 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	987 \| iso2022_jp \| csiso2022jp, iso2022jp, \| Japanese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	988 \| \| iso-2022-jp \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	989 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	990 \| iso2022_jp_1 \| iso2022jp-1, iso-2022-jp-1 \| Japanese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	991 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	992 \| iso2022_jp_2 \| iso2022jp-2, iso-2022-jp-2 \| Japanese, Korean, Simplified \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	993 \| \| \| Chinese, Western Europe, Greek \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	994 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	995 \| iso2022_jp_2004 \| iso2022jp-2004, \| Japanese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	996 \| \| iso-2022-jp-2004 \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	997 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	998 \| iso2022_jp_3 \| iso2022jp-3, iso-2022-jp-3 \| Japanese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	999 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1000 \| iso2022_jp_ext \| iso2022jp-ext, iso-2022-jp-ext \| Japanese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1001 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1002 \| iso2022_kr \| csiso2022kr, iso2022kr, \| Korean \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1003 \| \| iso-2022-kr \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1004 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1005 \| latin_1 \| iso-8859-1, iso8859-1, 8859, \| West Europe \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1006 \| \| cp819, latin, latin1, L1 \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1007 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1008 \| iso8859_2 \| iso-8859-2, latin2, L2 \| Central and Eastern Europe \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1009 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1010 \| iso8859_3 \| iso-8859-3, latin3, L3 \| Esperanto, Maltese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1011 +-----------------+--------------------------------+--------------------------------+
45429 853af5bbb497 A lot more typo fixes by Ori Avtalion. Georg Brandl <georg@python.org> parents: 45376 diff changeset	1012 \| iso8859_4 \| iso-8859-4, latin4, L4 \| Baltic languages \|
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1013 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1014 \| iso8859_5 \| iso-8859-5, cyrillic \| Bulgarian, Byelorussian, \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1015 \| \| \| Macedonian, Russian, Serbian \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1016 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1017 \| iso8859_6 \| iso-8859-6, arabic \| Arabic \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1018 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1019 \| iso8859_7 \| iso-8859-7, greek, greek8 \| Greek \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1020 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1021 \| iso8859_8 \| iso-8859-8, hebrew \| Hebrew \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1022 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1023 \| iso8859_9 \| iso-8859-9, latin5, L5 \| Turkish \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1024 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1025 \| iso8859_10 \| iso-8859-10, latin6, L6 \| Nordic languages \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1026 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1027 \| iso8859_13 \| iso-8859-13 \| Baltic languages \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1028 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1029 \| iso8859_14 \| iso-8859-14, latin8, L8 \| Celtic languages \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1030 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1031 \| iso8859_15 \| iso-8859-15 \| Western Europe \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1032 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1033 \| johab \| cp1361, ms1361 \| Korean \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1034 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1035 \| koi8_r \| \| Russian \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1036 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1037 \| koi8_u \| \| Ukrainian \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1038 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1039 \| mac_cyrillic \| maccyrillic \| Bulgarian, Byelorussian, \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1040 \| \| \| Macedonian, Russian, Serbian \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1041 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1042 \| mac_greek \| macgreek \| Greek \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1043 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1044 \| mac_iceland \| maciceland \| Icelandic \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1045 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1046 \| mac_latin2 \| maclatin2, maccentraleurope \| Central and Eastern Europe \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1047 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1048 \| mac_roman \| macroman \| Western Europe \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1049 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1050 \| mac_turkish \| macturkish \| Turkish \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1051 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1052 \| ptcp154 \| csptcp154, pt154, cp154, \| Kazakh \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1053 \| \| cyrillic-asian \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1054 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1055 \| shift_jis \| csshiftjis, shiftjis, sjis, \| Japanese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1056 \| \| s_jis \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1057 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1058 \| shift_jis_2004 \| shiftjis2004, sjis_2004, \| Japanese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1059 \| \| sjis2004 \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1060 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1061 \| shift_jisx0213 \| shiftjisx0213, sjisx0213, \| Japanese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1062 \| \| s_jisx0213 \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1063 +-----------------+--------------------------------+--------------------------------+
42649 c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 42603 diff changeset	1064 \| utf_32 \| U32, utf32 \| all languages \|
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 42603 diff changeset	1065 +-----------------+--------------------------------+--------------------------------+
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 42603 diff changeset	1066 \| utf_32_be \| UTF-32BE \| all languages \|
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 42603 diff changeset	1067 +-----------------+--------------------------------+--------------------------------+
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 42603 diff changeset	1068 \| utf_32_le \| UTF-32LE \| all languages \|
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 42603 diff changeset	1069 +-----------------+--------------------------------+--------------------------------+
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1070 \| utf_16 \| U16, utf16 \| all languages \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1071 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1072 \| utf_16_be \| UTF-16BE \| all languages (BMP only) \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1073 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1074 \| utf_16_le \| UTF-16LE \| all languages (BMP only) \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1075 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1076 \| utf_7 \| U7, unicode-1-1-utf-7 \| all languages \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1077 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1078 \| utf_8 \| U8, UTF, utf8 \| all languages \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1079 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1080 \| utf_8_sig \| \| all languages \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1081 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1082
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1083 A number of codecs are specific to Python, so their codec names have no meaning
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1084 outside Python. Some of them don't convert from Unicode strings to byte strings,
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1085 but instead use the property of the Python codecs machinery that any bijective
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1086 function with one argument can be considered as an encoding.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1087
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1088 For the codecs listed below, the result in the "encoding" direction is always a
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1089 byte string. The result of the "decoding" direction is listed as operand type in
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1090 the table.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1091
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1092 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1093 \| Codec \| Aliases \| Operand type \| Purpose \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1094 +====================+===========================+================+===========================+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1095 \| base64_codec \| base64, base-64 \| byte string \| Convert operand to MIME \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1096 \| \| \| \| base64 \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1097 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1098 \| bz2_codec \| bz2 \| byte string \| Compress the operand \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1099 \| \| \| \| using bz2 \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1100 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1101 \| hex_codec \| hex \| byte string \| Convert operand to \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1102 \| \| \| \| hexadecimal \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1103 \| \| \| \| representation, with two \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1104 \| \| \| \| digits per byte \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1105 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1106 \| idna \| \| Unicode string \| Implements :rfc:`3490`, \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1107 \| \| \| \| see also \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1108 \| \| \| \| :mod:`encodings.idna` \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1109 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1110 \| mbcs \| dbcs \| Unicode string \| Windows only: Encode \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1111 \| \| \| \| operand according to the \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1112 \| \| \| \| ANSI codepage (CP_ACP) \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1113 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1114 \| palmos \| \| Unicode string \| Encoding of PalmOS 3.5 \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1115 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1116 \| punycode \| \| Unicode string \| Implements :rfc:`3492` \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1117 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1118 \| quopri_codec \| quopri, quoted-printable, \| byte string \| Convert operand to MIME \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1119 \| \| quotedprintable \| \| quoted printable \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1120 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1121 \| raw_unicode_escape \| \| Unicode string \| Produce a string that is \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1122 \| \| \| \| suitable as raw Unicode \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1123 \| \| \| \| literal in Python source \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1124 \| \| \| \| code \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1125 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1126 \| rot_13 \| rot13 \| Unicode string \| Returns the Caesar-cypher \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1127 \| \| \| \| encryption of the operand \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1128 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1129 \| string_escape \| \| byte string \| Produce a string that is \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1130 \| \| \| \| suitable as string \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1131 \| \| \| \| literal in Python source \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1132 \| \| \| \| code \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1133 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1134 \| undefined \| \| any \| Raise an exception for \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1135 \| \| \| \| all conversions. Can be \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1136 \| \| \| \| used as the system \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1137 \| \| \| \| encoding if no automatic \|
44103 1e266015cf5b Add more entries to the glossary. Georg Brandl <georg@python.org> parents: 43629 diff changeset	1138 \| \| \| \| :term:`coercion` between \|
1e266015cf5b Add more entries to the glossary. Georg Brandl <georg@python.org> parents: 43629 diff changeset	1139 \| \| \| \| byte and Unicode strings \|
1e266015cf5b Add more entries to the glossary. Georg Brandl <georg@python.org> parents: 43629 diff changeset	1140 \| \| \| \| is desired. \|
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1141 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1142 \| unicode_escape \| \| Unicode string \| Produce a string that is \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1143 \| \| \| \| suitable as Unicode \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1144 \| \| \| \| literal in Python source \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1145 \| \| \| \| code \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1146 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1147 \| unicode_internal \| \| Unicode string \| Return the internal \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1148 \| \| \| \| representation of the \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1149 \| \| \| \| operand \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1150 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1151 \| uu_codec \| uu \| byte string \| Convert the operand using \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1152 \| \| \| \| uuencode \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1153 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1154 \| zlib_codec \| zip, zlib \| byte string \| Compress the operand \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1155 \| \| \| \| using gzip \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1156 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1157
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1158 .. versionadded:: 2.3
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1159 The ``idna`` and ``punycode`` encodings.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1160
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1161
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1162 :mod:`encodings.idna` --- Internationalized Domain Names in Applications
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1163 ------------------------------------------------------------------------
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1164
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1165 .. module:: encodings.idna
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1166 :synopsis: Internationalized Domain Names implementation
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1167 .. moduleauthor:: Martin v. Löwis
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1168
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1169 .. versionadded:: 2.3
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1170
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1171 This module implements :rfc:`3490` (Internationalized Domain Names in
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1172 Applications) and :rfc:`3492` (Nameprep: A Stringprep Profile for
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1173 Internationalized Domain Names (IDN)). It builds upon the ``punycode`` encoding
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1174 and :mod:`stringprep`.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1175
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1176 These RFCs together define a protocol to support non-ASCII characters in domain
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1177 names. A domain name containing non-ASCII characters (such as
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1178 ``www.Alliancefrançaise.nu``) is converted into an ASCII-compatible encoding
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1179 (ACE, such as ``www.xn--alliancefranaise-npb.nu``). The ACE form of the domain
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1180 name is then used in all places where arbitrary characters are not allowed by
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1181 the protocol, such as DNS queries, HTTP :mailheader:`Host` fields, and so
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1182 on. This conversion is carried out in the application; if possible invisible to
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1183 the user: The application should transparently convert Unicode domain labels to
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1184 IDNA on the wire, and convert back ACE labels to Unicode before presenting them
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1185 to the user.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1186
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1187 Python supports this conversion in several ways: The ``idna`` codec allows to
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1188 convert between Unicode and the ACE. Furthermore, the :mod:`socket` module
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1189 transparently converts Unicode host names to ACE, so that applications need not
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1190 be concerned about converting host names themselves when they pass them to the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1191 socket module. On top of that, modules that have host names as function
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1192 parameters, such as :mod:`httplib` and :mod:`ftplib`, accept Unicode host names
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1193 (:mod:`httplib` then also transparently sends an IDNA hostname in the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1194 :mailheader:`Host` field if it sends that field at all).
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1195
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1196 When receiving host names from the wire (such as in reverse name lookup), no
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1197 automatic conversion to Unicode is performed: Applications wishing to present
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1198 such host names to the user should decode them to Unicode.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1199
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1200 The module :mod:`encodings.idna` also implements the nameprep procedure, which
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1201 performs certain normalizations on host names, to achieve case-insensitivity of
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1202 international domain names, and to unify similar characters. The nameprep
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1203 functions can be used directly if desired.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1204
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1205
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1206 .. function:: nameprep(label)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1207
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1208 Return the nameprepped version of label. The implementation currently assumes
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1209 query strings, so ``AllowUnassigned`` is true.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1210
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1211
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1212 .. function:: ToASCII(label)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1213
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1214 Convert a label to ASCII, as specified in :rfc:`3490`. ``UseSTD3ASCIIRules`` is
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1215 assumed to be false.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1216
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1217
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1218 .. function:: ToUnicode(label)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1219
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1220 Convert a label to Unicode, as specified in :rfc:`3490`.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1221
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1222
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1223 :mod:`encodings.utf_8_sig` --- UTF-8 codec with BOM signature
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1224 -------------------------------------------------------------
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1225
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1226 .. module:: encodings.utf_8_sig
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1227 :synopsis: UTF-8 codec with BOM signature
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1228 .. moduleauthor:: Walter Dörwald
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1229
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1230 .. versionadded:: 2.5
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1231
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1232 This module implements a variant of the UTF-8 codec: On encoding a UTF-8 encoded
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1233 BOM will be prepended to the UTF-8 encoded bytes. For the stateful encoder this
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1234 is only done once (on the first write to the byte stream). For decoding an
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1235 optional UTF-8 encoded BOM at the start of the data will be skipped.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1236

Mercurial > cpython-lru_cache-weakref

annotate Doc/library/codecs.rst @ 54294:dba767bc3155 legacy-trunk