cpython-lru_cache-weakref: Doc/library/codecs.rst annotate

annotate Doc/library/codecs.rst @ 79675:c0a2294de787 2.7

Issue #16199: Backport 2to3 typo fix from 3.2.

author	Chris Jerdonek <chris.jerdonek@gmail.com>
date	Thu, 11 Oct 2012 16:15:46 -0700
parents	74ff71b2f53d
children	75512a165318

rev	line source
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	2 :mod:`codecs` --- Codec registry and base classes
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	3 =================================================
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	4
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	5 .. module:: codecs
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	6 :synopsis: Encode and decode data and streams.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	7 .. moduleauthor:: Marc-Andre Lemburg <mal@lemburg.com>
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	8 .. sectionauthor:: Marc-Andre Lemburg <mal@lemburg.com>
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	9 .. sectionauthor:: Martin v. Löwis <martin@v.loewis.de>
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	10
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	11
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	12 .. index::
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	13 single: Unicode
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	14 single: Codecs
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	15 pair: Codecs; encode
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	16 pair: Codecs; decode
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	17 single: streams
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	18 pair: stackable; streams
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	19
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	20 This module defines base classes for standard Python codecs (encoders and
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	21 decoders) and provides access to the internal Python codec registry which
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	22 manages the codec and error handling lookup process.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	23
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	24 It defines the following functions:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	25
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	26
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	27 .. function:: register(search_function)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	28
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	29 Register a codec search function. Search functions are expected to take one
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	30 argument, the encoding name in all lower case letters, and return a
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	31 :class:`CodecInfo` object having the following attributes:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	32
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	33 * ``name`` The name of the encoding;
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	34
49677 82ccc7e77ec5 Merged revisions 66801,66803-66804,66813,66854-66856,66866,66870-66872,66874,66887,66903,66905,66911,66913,66927,66932,66938,66942,66962,66964,66973-66974,66977,66992,66998-66999,67002,67005,67007,67028,67040-67041,67044,67070,67089,67091,67101,67117-67119,67123-67124 via svnmerge from Georg Brandl <georg@python.org> parents: 48919 diff changeset	35 * ``encode`` The stateless encoding function;
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	36
49677 82ccc7e77ec5 Merged revisions 66801,66803-66804,66813,66854-66856,66866,66870-66872,66874,66887,66903,66905,66911,66913,66927,66932,66938,66942,66962,66964,66973-66974,66977,66992,66998-66999,67002,67005,67007,67028,67040-67041,67044,67070,67089,67091,67101,67117-67119,67123-67124 via svnmerge from Georg Brandl <georg@python.org> parents: 48919 diff changeset	37 * ``decode`` The stateless decoding function;
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	38
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	39 * ``incrementalencoder`` An incremental encoder class or factory function;
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	40
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	41 * ``incrementaldecoder`` An incremental decoder class or factory function;
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	42
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	43 * ``streamwriter`` A stream writer class or factory function;
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	44
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	45 * ``streamreader`` A stream reader class or factory function.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	46
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	47 The various functions or classes take the following arguments:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	48
49677 82ccc7e77ec5 Merged revisions 66801,66803-66804,66813,66854-66856,66866,66870-66872,66874,66887,66903,66905,66911,66913,66927,66932,66938,66942,66962,66964,66973-66974,66977,66992,66998-66999,67002,67005,67007,67028,67040-67041,67044,67070,67089,67091,67101,67117-67119,67123-67124 via svnmerge from Georg Brandl <georg@python.org> parents: 48919 diff changeset	49 encode and decode: These must be functions or methods which have the same
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	50 interface as the :meth:`encode`/:meth:`decode` methods of Codec instances (see
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	51 Codec Interface). The functions/methods are expected to work in a stateless
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	52 mode.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	53
48919 13fdf98ea290 #3749: fix c'n'p errors. Georg Brandl <georg@python.org> parents: 47340 diff changeset	54 incrementalencoder and incrementaldecoder: These have to be factory
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	55 functions providing the following interface:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	56
56824 b7b8970cd5a5 Merged revisions 75363,75365,75376,75392,75394,75403,75418,75484,75572,75580,75590,75592,75594-75596,75600,75602-75603,75605-75607,75610-75613,75616-75617,75623,75627,75647 via svnmerge from Georg Brandl <georg@python.org> parents: 56822 diff changeset	57 ``factory(errors='strict')``
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	58
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	59 The factory functions must return objects providing the interfaces defined by
48919 13fdf98ea290 #3749: fix c'n'p errors. Georg Brandl <georg@python.org> parents: 47340 diff changeset	60 the base classes :class:`IncrementalEncoder` and :class:`IncrementalDecoder`,
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	61 respectively. Incremental codecs can maintain state.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	62
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	63 streamreader and streamwriter: These have to be factory functions providing
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	64 the following interface:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	65
56824 b7b8970cd5a5 Merged revisions 75363,75365,75376,75392,75394,75403,75418,75484,75572,75580,75590,75592,75594-75596,75600,75602-75603,75605-75607,75610-75613,75616-75617,75623,75627,75647 via svnmerge from Georg Brandl <georg@python.org> parents: 56822 diff changeset	66 ``factory(stream, errors='strict')``
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	67
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	68 The factory functions must return objects providing the interfaces defined by
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	69 the base classes :class:`StreamWriter` and :class:`StreamReader`, respectively.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	70 Stream codecs can maintain state.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	71
56824 b7b8970cd5a5 Merged revisions 75363,75365,75376,75392,75394,75403,75418,75484,75572,75580,75590,75592,75594-75596,75600,75602-75603,75605-75607,75610-75613,75616-75617,75623,75627,75647 via svnmerge from Georg Brandl <georg@python.org> parents: 56822 diff changeset	72 Possible values for errors are
b7b8970cd5a5 Merged revisions 75363,75365,75376,75392,75394,75403,75418,75484,75572,75580,75590,75592,75594-75596,75600,75602-75603,75605-75607,75610-75613,75616-75617,75623,75627,75647 via svnmerge from Georg Brandl <georg@python.org> parents: 56822 diff changeset	73
b7b8970cd5a5 Merged revisions 75363,75365,75376,75392,75394,75403,75418,75484,75572,75580,75590,75592,75594-75596,75600,75602-75603,75605-75607,75610-75613,75616-75617,75623,75627,75647 via svnmerge from Georg Brandl <georg@python.org> parents: 56822 diff changeset	74 * ``'strict'``: raise an exception in case of an encoding error
b7b8970cd5a5 Merged revisions 75363,75365,75376,75392,75394,75403,75418,75484,75572,75580,75590,75592,75594-75596,75600,75602-75603,75605-75607,75610-75613,75616-75617,75623,75627,75647 via svnmerge from Georg Brandl <georg@python.org> parents: 56822 diff changeset	75 * ``'replace'``: replace malformed data with a suitable replacement marker,
b7b8970cd5a5 Merged revisions 75363,75365,75376,75392,75394,75403,75418,75484,75572,75580,75590,75592,75594-75596,75600,75602-75603,75605-75607,75610-75613,75616-75617,75623,75627,75647 via svnmerge from Georg Brandl <georg@python.org> parents: 56822 diff changeset	76 such as ``'?'`` or ``'\ufffd'``
b7b8970cd5a5 Merged revisions 75363,75365,75376,75392,75394,75403,75418,75484,75572,75580,75590,75592,75594-75596,75600,75602-75603,75605-75607,75610-75613,75616-75617,75623,75627,75647 via svnmerge from Georg Brandl <georg@python.org> parents: 56822 diff changeset	77 * ``'ignore'``: ignore malformed data and continue without further notice
b7b8970cd5a5 Merged revisions 75363,75365,75376,75392,75394,75403,75418,75484,75572,75580,75590,75592,75594-75596,75600,75602-75603,75605-75607,75610-75613,75616-75617,75623,75627,75647 via svnmerge from Georg Brandl <georg@python.org> parents: 56822 diff changeset	78 * ``'xmlcharrefreplace'``: replace with the appropriate XML character
b7b8970cd5a5 Merged revisions 75363,75365,75376,75392,75394,75403,75418,75484,75572,75580,75590,75592,75594-75596,75600,75602-75603,75605-75607,75610-75613,75616-75617,75623,75627,75647 via svnmerge from Georg Brandl <georg@python.org> parents: 56822 diff changeset	79 reference (for encoding only)
b7b8970cd5a5 Merged revisions 75363,75365,75376,75392,75394,75403,75418,75484,75572,75580,75590,75592,75594-75596,75600,75602-75603,75605-75607,75610-75613,75616-75617,75623,75627,75647 via svnmerge from Georg Brandl <georg@python.org> parents: 56822 diff changeset	80 * ``'backslashreplace'``: replace with backslashed escape sequences (for
59176 f139ca034c3b Merged revisions 78489 via svnmerge from Ezio Melotti <ezio.melotti@gmail.com> parents: 56824 diff changeset	81 encoding only)
56824 b7b8970cd5a5 Merged revisions 75363,75365,75376,75392,75394,75403,75418,75484,75572,75580,75590,75592,75594-75596,75600,75602-75603,75605-75607,75610-75613,75616-75617,75623,75627,75647 via svnmerge from Georg Brandl <georg@python.org> parents: 56822 diff changeset	82
b7b8970cd5a5 Merged revisions 75363,75365,75376,75392,75394,75403,75418,75484,75572,75580,75590,75592,75594-75596,75600,75602-75603,75605-75607,75610-75613,75616-75617,75623,75627,75647 via svnmerge from Georg Brandl <georg@python.org> parents: 56822 diff changeset	83 as well as any other error handling name defined via :func:`register_error`.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	84
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	85 In case a search function cannot find a given encoding, it should return
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	86 ``None``.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	87
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	88
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	89 .. function:: lookup(encoding)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	90
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	91 Looks up the codec info in the Python codec registry and returns a
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	92 :class:`CodecInfo` object as defined above.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	93
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	94 Encodings are first looked up in the registry's cache. If not found, the list of
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	95 registered search functions is scanned. If no :class:`CodecInfo` object is
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	96 found, a :exc:`LookupError` is raised. Otherwise, the :class:`CodecInfo` object
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	97 is stored in the cache and returned to the caller.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	98
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	99 To simplify access to the various codecs, the module provides these additional
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	100 functions which use :func:`lookup` for the codec lookup:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	101
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	102
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	103 .. function:: getencoder(encoding)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	104
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	105 Look up the codec for the given encoding and return its encoder function.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	106
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	107 Raises a :exc:`LookupError` in case the encoding cannot be found.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	108
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	109
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	110 .. function:: getdecoder(encoding)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	111
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	112 Look up the codec for the given encoding and return its decoder function.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	113
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	114 Raises a :exc:`LookupError` in case the encoding cannot be found.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	115
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	116
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	117 .. function:: getincrementalencoder(encoding)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	118
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	119 Look up the codec for the given encoding and return its incremental encoder
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	120 class or factory function.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	121
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	122 Raises a :exc:`LookupError` in case the encoding cannot be found or the codec
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	123 doesn't support an incremental encoder.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	124
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	125 .. versionadded:: 2.5
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	126
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	127
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	128 .. function:: getincrementaldecoder(encoding)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	129
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	130 Look up the codec for the given encoding and return its incremental decoder
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	131 class or factory function.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	132
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	133 Raises a :exc:`LookupError` in case the encoding cannot be found or the codec
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	134 doesn't support an incremental decoder.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	135
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	136 .. versionadded:: 2.5
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	137
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	138
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	139 .. function:: getreader(encoding)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	140
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	141 Look up the codec for the given encoding and return its StreamReader class or
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	142 factory function.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	143
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	144 Raises a :exc:`LookupError` in case the encoding cannot be found.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	145
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	146
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	147 .. function:: getwriter(encoding)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	148
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	149 Look up the codec for the given encoding and return its StreamWriter class or
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	150 factory function.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	151
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	152 Raises a :exc:`LookupError` in case the encoding cannot be found.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	153
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	154
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	155 .. function:: register_error(name, error_handler)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	156
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	157 Register the error handling function error_handler under the name name.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	158 error_handler will be called during encoding and decoding in case of an error,
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	159 when name is specified as the errors parameter.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	160
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	161 For encoding error_handler will be called with a :exc:`UnicodeEncodeError`
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	162 instance, which contains information about the location of the error. The error
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	163 handler must either raise this or a different exception or return a tuple with a
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	164 replacement for the unencodable part of the input and a position where encoding
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	165 should continue. The encoder will encode the replacement and continue encoding
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	166 the original input at the specified position. Negative position values will be
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	167 treated as being relative to the end of the input string. If the resulting
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	168 position is out of bound an :exc:`IndexError` will be raised.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	169
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	170 Decoding and translating works similar, except :exc:`UnicodeDecodeError` or
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	171 :exc:`UnicodeTranslateError` will be passed to the handler and that the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	172 replacement from the error handler will be put into the output directly.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	173
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	174
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	175 .. function:: lookup_error(name)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	176
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	177 Return the error handler previously registered under the name name.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	178
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	179 Raises a :exc:`LookupError` in case the handler cannot be found.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	180
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	181
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	182 .. function:: strict_errors(exception)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	183
56824 b7b8970cd5a5 Merged revisions 75363,75365,75376,75392,75394,75403,75418,75484,75572,75580,75590,75592,75594-75596,75600,75602-75603,75605-75607,75610-75613,75616-75617,75623,75627,75647 via svnmerge from Georg Brandl <georg@python.org> parents: 56822 diff changeset	184 Implements the ``strict`` error handling: each encoding or decoding error
b7b8970cd5a5 Merged revisions 75363,75365,75376,75392,75394,75403,75418,75484,75572,75580,75590,75592,75594-75596,75600,75602-75603,75605-75607,75610-75613,75616-75617,75623,75627,75647 via svnmerge from Georg Brandl <georg@python.org> parents: 56822 diff changeset	185 raises a :exc:`UnicodeError`.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	186
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	187
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	188 .. function:: replace_errors(exception)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	189
56824 b7b8970cd5a5 Merged revisions 75363,75365,75376,75392,75394,75403,75418,75484,75572,75580,75590,75592,75594-75596,75600,75602-75603,75605-75607,75610-75613,75616-75617,75623,75627,75647 via svnmerge from Georg Brandl <georg@python.org> parents: 56822 diff changeset	190 Implements the ``replace`` error handling: malformed data is replaced with a
b7b8970cd5a5 Merged revisions 75363,75365,75376,75392,75394,75403,75418,75484,75572,75580,75590,75592,75594-75596,75600,75602-75603,75605-75607,75610-75613,75616-75617,75623,75627,75647 via svnmerge from Georg Brandl <georg@python.org> parents: 56822 diff changeset	191 suitable replacement character such as ``'?'`` in bytestrings and
b7b8970cd5a5 Merged revisions 75363,75365,75376,75392,75394,75403,75418,75484,75572,75580,75590,75592,75594-75596,75600,75602-75603,75605-75607,75610-75613,75616-75617,75623,75627,75647 via svnmerge from Georg Brandl <georg@python.org> parents: 56822 diff changeset	192 ``'\ufffd'`` in Unicode strings.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	193
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	194
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	195 .. function:: ignore_errors(exception)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	196
56824 b7b8970cd5a5 Merged revisions 75363,75365,75376,75392,75394,75403,75418,75484,75572,75580,75590,75592,75594-75596,75600,75602-75603,75605-75607,75610-75613,75616-75617,75623,75627,75647 via svnmerge from Georg Brandl <georg@python.org> parents: 56822 diff changeset	197 Implements the ``ignore`` error handling: malformed data is ignored and
b7b8970cd5a5 Merged revisions 75363,75365,75376,75392,75394,75403,75418,75484,75572,75580,75590,75592,75594-75596,75600,75602-75603,75605-75607,75610-75613,75616-75617,75623,75627,75647 via svnmerge from Georg Brandl <georg@python.org> parents: 56822 diff changeset	198 encoding or decoding is continued without further notice.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	199
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	200
43184 ad3d5acacf03 Fix wrong function names. Walter Dörwald <walter@livinglogic.de> parents: 42649 diff changeset	201 .. function:: xmlcharrefreplace_errors(exception)
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	202
56824 b7b8970cd5a5 Merged revisions 75363,75365,75376,75392,75394,75403,75418,75484,75572,75580,75590,75592,75594-75596,75600,75602-75603,75605-75607,75610-75613,75616-75617,75623,75627,75647 via svnmerge from Georg Brandl <georg@python.org> parents: 56822 diff changeset	203 Implements the ``xmlcharrefreplace`` error handling (for encoding only): the
b7b8970cd5a5 Merged revisions 75363,75365,75376,75392,75394,75403,75418,75484,75572,75580,75590,75592,75594-75596,75600,75602-75603,75605-75607,75610-75613,75616-75617,75623,75627,75647 via svnmerge from Georg Brandl <georg@python.org> parents: 56822 diff changeset	204 unencodable character is replaced by an appropriate XML character reference.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	205
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	206
43184 ad3d5acacf03 Fix wrong function names. Walter Dörwald <walter@livinglogic.de> parents: 42649 diff changeset	207 .. function:: backslashreplace_errors(exception)
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	208
56824 b7b8970cd5a5 Merged revisions 75363,75365,75376,75392,75394,75403,75418,75484,75572,75580,75590,75592,75594-75596,75600,75602-75603,75605-75607,75610-75613,75616-75617,75623,75627,75647 via svnmerge from Georg Brandl <georg@python.org> parents: 56822 diff changeset	209 Implements the ``backslashreplace`` error handling (for encoding only): the
b7b8970cd5a5 Merged revisions 75363,75365,75376,75392,75394,75403,75418,75484,75572,75580,75590,75592,75594-75596,75600,75602-75603,75605-75607,75610-75613,75616-75617,75623,75627,75647 via svnmerge from Georg Brandl <georg@python.org> parents: 56822 diff changeset	210 unencodable character is replaced by a backslashed escape sequence.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	211
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	212 To simplify working with encoded files or stream, the module also defines these
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	213 utility functions:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	214
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	215
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	216 .. function:: open(filename, mode[, encoding[, errors[, buffering]]])
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	217
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	218 Open an encoded file using the given mode and return a wrapped version
45376 6212367390e3 #2131: note that codecs.open() always opens files in binary mode. Georg Brandl <georg@python.org> parents: 44103 diff changeset	219 providing transparent encoding/decoding. The default file mode is ``'r'``
6212367390e3 #2131: note that codecs.open() always opens files in binary mode. Georg Brandl <georg@python.org> parents: 44103 diff changeset	220 meaning to open the file in read mode.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	221
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	222 .. note::
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	223
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	224 The wrapped version will only accept the object format defined by the codecs,
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	225 i.e. Unicode objects for most built-in codecs. Output is also codec-dependent
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	226 and will usually be Unicode as well.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	227
45376 6212367390e3 #2131: note that codecs.open() always opens files in binary mode. Georg Brandl <georg@python.org> parents: 44103 diff changeset	228 .. note::
6212367390e3 #2131: note that codecs.open() always opens files in binary mode. Georg Brandl <georg@python.org> parents: 44103 diff changeset	229
6212367390e3 #2131: note that codecs.open() always opens files in binary mode. Georg Brandl <georg@python.org> parents: 44103 diff changeset	230 Files are always opened in binary mode, even if no binary mode was
6212367390e3 #2131: note that codecs.open() always opens files in binary mode. Georg Brandl <georg@python.org> parents: 44103 diff changeset	231 specified. This is done to avoid data loss due to encodings using 8-bit
6212367390e3 #2131: note that codecs.open() always opens files in binary mode. Georg Brandl <georg@python.org> parents: 44103 diff changeset	232 values. This means that no automatic conversion of ``'\n'`` is done
6212367390e3 #2131: note that codecs.open() always opens files in binary mode. Georg Brandl <georg@python.org> parents: 44103 diff changeset	233 on reading and writing.
6212367390e3 #2131: note that codecs.open() always opens files in binary mode. Georg Brandl <georg@python.org> parents: 44103 diff changeset	234
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	235 encoding specifies the encoding which is to be used for the file.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	236
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	237 errors may be given to define the error handling. It defaults to ``'strict'``
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	238 which causes a :exc:`ValueError` to be raised in case an encoding error occurs.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	239
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	240 buffering has the same meaning as for the built-in :func:`open` function. It
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	241 defaults to line buffered.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	242
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	243
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	244 .. function:: EncodedFile(file, input[, output[, errors]])
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	245
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	246 Return a wrapped version of file which provides transparent encoding
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	247 translation.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	248
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	249 Strings written to the wrapped file are interpreted according to the given
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	250 input encoding and then written to the original file as strings using the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	251 output encoding. The intermediate encoding will usually be Unicode but depends
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	252 on the specified codecs.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	253
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	254 If output is not given, it defaults to input.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	255
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	256 errors may be given to define the error handling. It defaults to ``'strict'``,
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	257 which causes :exc:`ValueError` to be raised in case an encoding error occurs.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	258
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	259
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	260 .. function:: iterencode(iterable, encoding[, errors])
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	261
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	262 Uses an incremental encoder to iteratively encode the input provided by
43629 4f662ba09f1f Add :term: for generators. Georg Brandl <georg@python.org> parents: 43185 diff changeset	263 iterable. This function is a :term:`generator`. errors (as well as any
4f662ba09f1f Add :term: for generators. Georg Brandl <georg@python.org> parents: 43185 diff changeset	264 other keyword argument) is passed through to the incremental encoder.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	265
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	266 .. versionadded:: 2.5
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	267
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	268
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	269 .. function:: iterdecode(iterable, encoding[, errors])
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	270
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	271 Uses an incremental decoder to iteratively decode the input provided by
43629 4f662ba09f1f Add :term: for generators. Georg Brandl <georg@python.org> parents: 43185 diff changeset	272 iterable. This function is a :term:`generator`. errors (as well as any
4f662ba09f1f Add :term: for generators. Georg Brandl <georg@python.org> parents: 43185 diff changeset	273 other keyword argument) is passed through to the incremental decoder.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	274
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	275 .. versionadded:: 2.5
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	276
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	277 The module also provides the following constants which are useful for reading
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	278 and writing to platform dependent files:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	279
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	280
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	281 .. data:: BOM
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	282 BOM_BE
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	283 BOM_LE
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	284 BOM_UTF8
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	285 BOM_UTF16
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	286 BOM_UTF16_BE
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	287 BOM_UTF16_LE
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	288 BOM_UTF32
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	289 BOM_UTF32_BE
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	290 BOM_UTF32_LE
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	291
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	292 These constants define various encodings of the Unicode byte order mark (BOM)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	293 used in UTF-16 and UTF-32 data streams to indicate the byte order used in the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	294 stream or file and in UTF-8 as a Unicode signature. :const:`BOM_UTF16` is either
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	295 :const:`BOM_UTF16_BE` or :const:`BOM_UTF16_LE` depending on the platform's
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	296 native byte order, :const:`BOM` is an alias for :const:`BOM_UTF16`,
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	297 :const:`BOM_LE` for :const:`BOM_UTF16_LE` and :const:`BOM_BE` for
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	298 :const:`BOM_UTF16_BE`. The others represent the BOM in UTF-8 and UTF-32
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	299 encodings.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	300
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	301
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	302 .. _codec-base-classes:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	303
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	304 Codec Base Classes
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	305 ------------------
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	306
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	307 The :mod:`codecs` module defines a set of base classes which define the
47340 f6c88f172bbe fix minor grammar typo Benjamin Peterson <benjamin@python.org> parents: 46555 diff changeset	308 interface and can also be used to easily write your own codecs for use in
f6c88f172bbe fix minor grammar typo Benjamin Peterson <benjamin@python.org> parents: 46555 diff changeset	309 Python.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	310
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	311 Each codec has to define four interfaces to make it usable as codec in Python:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	312 stateless encoder, stateless decoder, stream reader and stream writer. The
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	313 stream reader and writers typically reuse the stateless encoder/decoder to
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	314 implement the file protocols.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	315
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	316 The :class:`Codec` class defines the interface for stateless encoders/decoders.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	317
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	318 To simplify and standardize error handling, the :meth:`encode` and
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	319 :meth:`decode` methods may implement different error handling schemes by
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	320 providing the errors string argument. The following string values are defined
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	321 and implemented by all standard Python codecs:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	322
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	323 +-------------------------+-----------------------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	324 \| Value \| Meaning \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	325 +=========================+===============================================+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	326 \| ``'strict'`` \| Raise :exc:`UnicodeError` (or a subclass); \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	327 \| \| this is the default. \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	328 +-------------------------+-----------------------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	329 \| ``'ignore'`` \| Ignore the character and continue with the \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	330 \| \| next. \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	331 +-------------------------+-----------------------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	332 \| ``'replace'`` \| Replace with a suitable replacement \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	333 \| \| character; Python will use the official \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	334 \| \| U+FFFD REPLACEMENT CHARACTER for the built-in \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	335 \| \| Unicode codecs on decoding and '?' on \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	336 \| \| encoding. \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	337 +-------------------------+-----------------------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	338 \| ``'xmlcharrefreplace'`` \| Replace with the appropriate XML character \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	339 \| \| reference (only for encoding). \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	340 +-------------------------+-----------------------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	341 \| ``'backslashreplace'`` \| Replace with backslashed escape sequences \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	342 \| \| (only for encoding). \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	343 +-------------------------+-----------------------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	344
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	345 The set of allowed values can be extended via :meth:`register_error`.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	346
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	347
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	348 .. _codec-objects:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	349
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	350 Codec Objects
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	351 ^^^^^^^^^^^^^
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	352
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	353 The :class:`Codec` class defines these methods which also define the function
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	354 interfaces of the stateless encoder and decoder:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	355
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	356
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	357 .. method:: Codec.encode(input[, errors])
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	358
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	359 Encodes the object input and returns a tuple (output object, length consumed).
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	360 While codecs are not restricted to use with Unicode, in a Unicode context,
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	361 encoding converts a Unicode object to a plain string using a particular
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	362 character set encoding (e.g., ``cp1252`` or ``iso-8859-1``).
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	363
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	364 errors defines the error handling to apply. It defaults to ``'strict'``
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	365 handling.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	366
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	367 The method may not store state in the :class:`Codec` instance. Use
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	368 :class:`StreamCodec` for codecs which have to keep state in order to make
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	369 encoding/decoding efficient.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	370
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	371 The encoder must be able to handle zero length input and return an empty object
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	372 of the output object type in this situation.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	373
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	374
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	375 .. method:: Codec.decode(input[, errors])
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	376
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	377 Decodes the object input and returns a tuple (output object, length consumed).
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	378 In a Unicode context, decoding converts a plain string encoded using a
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	379 particular character set encoding to a Unicode object.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	380
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	381 input must be an object which provides the ``bf_getreadbuf`` buffer slot.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	382 Python strings, buffer objects and memory mapped files are examples of objects
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	383 providing this slot.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	384
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	385 errors defines the error handling to apply. It defaults to ``'strict'``
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	386 handling.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	387
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	388 The method may not store state in the :class:`Codec` instance. Use
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	389 :class:`StreamCodec` for codecs which have to keep state in order to make
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	390 encoding/decoding efficient.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	391
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	392 The decoder must be able to handle zero length input and return an empty object
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	393 of the output object type in this situation.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	394
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	395 The :class:`IncrementalEncoder` and :class:`IncrementalDecoder` classes provide
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	396 the basic interface for incremental encoding and decoding. Encoding/decoding the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	397 input isn't done with one call to the stateless encoder/decoder function, but
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	398 with multiple calls to the :meth:`encode`/:meth:`decode` method of the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	399 incremental encoder/decoder. The incremental encoder/decoder keeps track of the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	400 encoding/decoding process during method calls.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	401
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	402 The joined output of calls to the :meth:`encode`/:meth:`decode` method is the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	403 same as if all the single inputs were joined into one, and this input was
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	404 encoded/decoded with the stateless encoder/decoder.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	405
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	406
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	407 .. _incremental-encoder-objects:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	408
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	409 IncrementalEncoder Objects
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	410 ^^^^^^^^^^^^^^^^^^^^^^^^^^
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	411
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	412 .. versionadded:: 2.5
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	413
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	414 The :class:`IncrementalEncoder` class is used for encoding an input in multiple
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	415 steps. It defines the following methods which every incremental encoder must
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	416 define in order to be compatible with the Python codec registry.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	417
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	418
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	419 .. class:: IncrementalEncoder([errors])
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	420
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	421 Constructor for an :class:`IncrementalEncoder` instance.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	422
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	423 All incremental encoders must provide this constructor interface. They are free
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	424 to add additional keyword arguments, but only the ones defined here are used by
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	425 the Python codec registry.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	426
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	427 The :class:`IncrementalEncoder` may implement different error handling schemes
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	428 by providing the errors keyword argument. These parameters are predefined:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	429
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	430 * ``'strict'`` Raise :exc:`ValueError` (or a subclass); this is the default.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	431
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	432 * ``'ignore'`` Ignore the character and continue with the next.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	433
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	434 * ``'replace'`` Replace with a suitable replacement character
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	435
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	436 * ``'xmlcharrefreplace'`` Replace with the appropriate XML character reference
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	437
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	438 * ``'backslashreplace'`` Replace with backslashed escape sequences.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	439
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	440 The errors argument will be assigned to an attribute of the same name.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	441 Assigning to this attribute makes it possible to switch between different error
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	442 handling strategies during the lifetime of the :class:`IncrementalEncoder`
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	443 object.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	444
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	445 The set of allowed values for the errors argument can be extended with
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	446 :func:`register_error`.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	447
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	448
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	449 .. method:: encode(object[, final])
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	450
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	451 Encodes object (taking the current state of the encoder into account)
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	452 and returns the resulting encoded object. If this is the last call to
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	453 :meth:`encode` final must be true (the default is false).
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	454
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	455
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	456 .. method:: reset()
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	457
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	458 Reset the encoder to the initial state.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	459
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	460
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	461 .. _incremental-decoder-objects:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	462
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	463 IncrementalDecoder Objects
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	464 ^^^^^^^^^^^^^^^^^^^^^^^^^^
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	465
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	466 The :class:`IncrementalDecoder` class is used for decoding an input in multiple
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	467 steps. It defines the following methods which every incremental decoder must
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	468 define in order to be compatible with the Python codec registry.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	469
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	470
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	471 .. class:: IncrementalDecoder([errors])
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	472
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	473 Constructor for an :class:`IncrementalDecoder` instance.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	474
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	475 All incremental decoders must provide this constructor interface. They are free
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	476 to add additional keyword arguments, but only the ones defined here are used by
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	477 the Python codec registry.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	478
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	479 The :class:`IncrementalDecoder` may implement different error handling schemes
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	480 by providing the errors keyword argument. These parameters are predefined:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	481
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	482 * ``'strict'`` Raise :exc:`ValueError` (or a subclass); this is the default.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	483
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	484 * ``'ignore'`` Ignore the character and continue with the next.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	485
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	486 * ``'replace'`` Replace with a suitable replacement character.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	487
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	488 The errors argument will be assigned to an attribute of the same name.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	489 Assigning to this attribute makes it possible to switch between different error
48919 13fdf98ea290 #3749: fix c'n'p errors. Georg Brandl <georg@python.org> parents: 47340 diff changeset	490 handling strategies during the lifetime of the :class:`IncrementalDecoder`
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	491 object.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	492
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	493 The set of allowed values for the errors argument can be extended with
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	494 :func:`register_error`.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	495
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	496
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	497 .. method:: decode(object[, final])
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	498
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	499 Decodes object (taking the current state of the decoder into account)
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	500 and returns the resulting decoded object. If this is the last call to
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	501 :meth:`decode` final must be true (the default is false). If final is
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	502 true the decoder must decode the input completely and must flush all
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	503 buffers. If this isn't possible (e.g. because of incomplete byte sequences
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	504 at the end of the input) it must initiate error handling just like in the
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	505 stateless case (which might raise an exception).
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	506
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	507
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	508 .. method:: reset()
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	509
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	510 Reset the decoder to the initial state.
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	511
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	512
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	513 The :class:`StreamWriter` and :class:`StreamReader` classes provide generic
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	514 working interfaces which can be used to implement new encoding submodules very
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	515 easily. See :mod:`encodings.utf_8` for an example of how this is done.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	516
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	517
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	518 .. _stream-writer-objects:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	519
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	520 StreamWriter Objects
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	521 ^^^^^^^^^^^^^^^^^^^^
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	522
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	523 The :class:`StreamWriter` class is a subclass of :class:`Codec` and defines the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	524 following methods which every stream writer must define in order to be
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	525 compatible with the Python codec registry.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	526
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	527
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	528 .. class:: StreamWriter(stream[, errors])
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	529
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	530 Constructor for a :class:`StreamWriter` instance.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	531
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	532 All stream writers must provide this constructor interface. They are free to add
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	533 additional keyword arguments, but only the ones defined here are used by the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	534 Python codec registry.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	535
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	536 stream must be a file-like object open for writing binary data.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	537
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	538 The :class:`StreamWriter` may implement different error handling schemes by
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	539 providing the errors keyword argument. These parameters are predefined:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	540
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	541 * ``'strict'`` Raise :exc:`ValueError` (or a subclass); this is the default.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	542
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	543 * ``'ignore'`` Ignore the character and continue with the next.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	544
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	545 * ``'replace'`` Replace with a suitable replacement character
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	546
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	547 * ``'xmlcharrefreplace'`` Replace with the appropriate XML character reference
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	548
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	549 * ``'backslashreplace'`` Replace with backslashed escape sequences.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	550
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	551 The errors argument will be assigned to an attribute of the same name.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	552 Assigning to this attribute makes it possible to switch between different error
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	553 handling strategies during the lifetime of the :class:`StreamWriter` object.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	554
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	555 The set of allowed values for the errors argument can be extended with
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	556 :func:`register_error`.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	557
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	558
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	559 .. method:: write(object)
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	560
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	561 Writes the object's contents encoded to the stream.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	562
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	563
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	564 .. method:: writelines(list)
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	565
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	566 Writes the concatenated list of strings to the stream (possibly by reusing
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	567 the :meth:`write` method).
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	568
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	569
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	570 .. method:: reset()
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	571
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	572 Flushes and resets the codec buffers used for keeping state.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	573
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	574 Calling this method should ensure that the data on the output is put into
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	575 a clean state that allows appending of new fresh data without having to
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	576 rescan the whole stream to recover state.
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	577
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	578
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	579 In addition to the above methods, the :class:`StreamWriter` must also inherit
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	580 all other methods and attributes from the underlying stream.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	581
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	582
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	583 .. _stream-reader-objects:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	584
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	585 StreamReader Objects
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	586 ^^^^^^^^^^^^^^^^^^^^
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	587
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	588 The :class:`StreamReader` class is a subclass of :class:`Codec` and defines the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	589 following methods which every stream reader must define in order to be
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	590 compatible with the Python codec registry.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	591
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	592
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	593 .. class:: StreamReader(stream[, errors])
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	594
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	595 Constructor for a :class:`StreamReader` instance.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	596
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	597 All stream readers must provide this constructor interface. They are free to add
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	598 additional keyword arguments, but only the ones defined here are used by the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	599 Python codec registry.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	600
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	601 stream must be a file-like object open for reading (binary) data.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	602
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	603 The :class:`StreamReader` may implement different error handling schemes by
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	604 providing the errors keyword argument. These parameters are defined:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	605
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	606 * ``'strict'`` Raise :exc:`ValueError` (or a subclass); this is the default.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	607
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	608 * ``'ignore'`` Ignore the character and continue with the next.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	609
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	610 * ``'replace'`` Replace with a suitable replacement character.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	611
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	612 The errors argument will be assigned to an attribute of the same name.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	613 Assigning to this attribute makes it possible to switch between different error
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	614 handling strategies during the lifetime of the :class:`StreamReader` object.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	615
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	616 The set of allowed values for the errors argument can be extended with
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	617 :func:`register_error`.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	618
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	619
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	620 .. method:: read([size[, chars, [firstline]]])
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	621
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	622 Decodes data from the stream and returns the resulting object.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	623
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	624 chars indicates the number of characters to read from the
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	625 stream. :func:`read` will never return more than chars characters, but
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	626 it might return less, if there are not enough characters available.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	627
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	628 size indicates the approximate maximum number of bytes to read from the
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	629 stream for decoding purposes. The decoder can modify this setting as
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	630 appropriate. The default value -1 indicates to read and decode as much as
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	631 possible. size is intended to prevent having to decode huge files in
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	632 one step.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	633
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	634 firstline indicates that it would be sufficient to only return the first
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	635 line, if there are decoding errors on later lines.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	636
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	637 The method should use a greedy read strategy meaning that it should read
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	638 as much data as is allowed within the definition of the encoding and the
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	639 given size, e.g. if optional encoding endings or state markers are
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	640 available on the stream, these should be read too.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	641
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	642 .. versionchanged:: 2.4
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	643 chars argument added.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	644
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	645 .. versionchanged:: 2.4.2
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	646 firstline argument added.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	647
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	648
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	649 .. method:: readline([size[, keepends]])
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	650
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	651 Read one line from the input stream and return the decoded data.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	652
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	653 size, if given, is passed as size argument to the stream's
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	654 :meth:`readline` method.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	655
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	656 If keepends is false line-endings will be stripped from the lines
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	657 returned.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	658
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	659 .. versionchanged:: 2.4
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	660 keepends argument added.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	661
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	662
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	663 .. method:: readlines([sizehint[, keepends]])
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	664
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	665 Read all lines available on the input stream and return them as a list of
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	666 lines.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	667
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	668 Line-endings are implemented using the codec's decoder method and are
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	669 included in the list entries if keepends is true.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	670
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	671 sizehint, if given, is passed as the size argument to the stream's
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	672 :meth:`read` method.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	673
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	674
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	675 .. method:: reset()
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	676
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	677 Resets the codec buffers used for keeping state.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	678
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	679 Note that no stream repositioning should take place. This method is
66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	680 primarily intended to be able to recover from decoding errors.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	681
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	682
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	683 In addition to the above methods, the :class:`StreamReader` must also inherit
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	684 all other methods and attributes from the underlying stream.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	685
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	686 The next two base classes are included for convenience. They are not needed by
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	687 the codec registry, but may provide useful in practice.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	688
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	689
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	690 .. _stream-reader-writer:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	691
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	692 StreamReaderWriter Objects
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	693 ^^^^^^^^^^^^^^^^^^^^^^^^^^
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	694
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	695 The :class:`StreamReaderWriter` allows wrapping streams which work in both read
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	696 and write modes.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	697
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	698 The design is such that one can use the factory functions returned by the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	699 :func:`lookup` function to construct the instance.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	700
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	701
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	702 .. class:: StreamReaderWriter(stream, Reader, Writer, errors)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	703
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	704 Creates a :class:`StreamReaderWriter` instance. stream must be a file-like
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	705 object. Reader and Writer must be factory functions or classes providing the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	706 :class:`StreamReader` and :class:`StreamWriter` interface resp. Error handling
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	707 is done in the same way as defined for the stream readers and writers.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	708
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	709 :class:`StreamReaderWriter` instances define the combined interfaces of
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	710 :class:`StreamReader` and :class:`StreamWriter` classes. They inherit all other
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	711 methods and attributes from the underlying stream.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	712
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	713
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	714 .. _stream-recoder-objects:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	715
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	716 StreamRecoder Objects
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	717 ^^^^^^^^^^^^^^^^^^^^^
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	718
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	719 The :class:`StreamRecoder` provide a frontend - backend view of encoding data
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	720 which is sometimes useful when dealing with different encoding environments.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	721
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	722 The design is such that one can use the factory functions returned by the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	723 :func:`lookup` function to construct the instance.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	724
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	725
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	726 .. class:: StreamRecoder(stream, encode, decode, Reader, Writer, errors)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	727
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	728 Creates a :class:`StreamRecoder` instance which implements a two-way conversion:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	729 encode and decode work on the frontend (the input to :meth:`read` and output
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	730 of :meth:`write`) while Reader and Writer work on the backend (reading and
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	731 writing to the stream).
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	732
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	733 You can use these objects to do transparent direct recodings from e.g. Latin-1
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	734 to UTF-8 and back.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	735
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	736 stream must be a file-like object.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	737
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	738 encode, decode must adhere to the :class:`Codec` interface. Reader,
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	739 Writer must be factory functions or classes providing objects of the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	740 :class:`StreamReader` and :class:`StreamWriter` interface respectively.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	741
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	742 encode and decode are needed for the frontend translation, Reader and
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	743 Writer for the backend translation. The intermediate format used is
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	744 determined by the two sets of codecs, e.g. the Unicode codecs will use Unicode
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	745 as the intermediate encoding.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	746
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	747 Error handling is done in the same way as defined for the stream readers and
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	748 writers.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	749
46555 66c8bd1bedf3 reformat some documentation of classes so methods and attributes are under the class directive Benjamin Peterson <benjamin@python.org> parents: 45429 diff changeset	750
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	751 :class:`StreamRecoder` instances define the combined interfaces of
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	752 :class:`StreamReader` and :class:`StreamWriter` classes. They inherit all other
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	753 methods and attributes from the underlying stream.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	754
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	755
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	756 .. _encodings-overview:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	757
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	758 Encodings and Unicode
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	759 ---------------------
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	760
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	761 Unicode strings are stored internally as sequences of codepoints (to be precise
74395 74ff71b2f53d update to new C roles and directives Sandro Tosi <sandro.tosi@gmail.com> parents: 73124 diff changeset	762 as :c:type:`Py_UNICODE` arrays). Depending on the way Python is compiled (either
66962 59e230297bff Merged revisions 86521,86632,86823-86824,87294,87296,87300,87302 via svnmerge from Éric Araujo <merwok@netwok.org> parents: 61884 diff changeset	763 via ``--enable-unicode=ucs2`` or ``--enable-unicode=ucs4``, with the
74395 74ff71b2f53d update to new C roles and directives Sandro Tosi <sandro.tosi@gmail.com> parents: 73124 diff changeset	764 former being the default) :c:type:`Py_UNICODE` is either a 16-bit or 32-bit data
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	765 type. Once a Unicode object is used outside of CPU and memory, CPU endianness
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	766 and how these arrays are stored as bytes become an issue. Transforming a
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	767 unicode object into a sequence of bytes is called encoding and recreating the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	768 unicode object from the sequence of bytes is known as decoding. There are many
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	769 different methods for how this transformation can be done (these methods are
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	770 also called encodings). The simplest method is to map the codepoints 0-255 to
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	771 the bytes ``0x0``-``0xff``. This means that a unicode object that contains
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	772 codepoints above ``U+00FF`` can't be encoded with this method (which is called
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	773 ``'latin-1'`` or ``'iso-8859-1'``). :func:`unicode.encode` will raise a
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	774 :exc:`UnicodeEncodeError` that looks like this: ``UnicodeEncodeError: 'latin-1'
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	775 codec can't encode character u'\u1234' in position 3: ordinal not in
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	776 range(256)``.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	777
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	778 There's another group of encodings (the so called charmap encodings) that choose
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	779 a different subset of all unicode code points and how these codepoints are
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	780 mapped to the bytes ``0x0``-``0xff``. To see how this is done simply open
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	781 e.g. :file:`encodings/cp1252.py` (which is an encoding that is used primarily on
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	782 Windows). There's a string constant with 256 characters that shows you which
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	783 character is mapped to which byte value.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	784
73124 6cfee8de0dea Refactor a bit the codecs doc. Ezio Melotti <ezio.melotti@gmail.com> parents: 72162 diff changeset	785 All of these encodings can only encode 256 of the 1114112 codepoints
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	786 defined in unicode. A simple and straightforward way that can store each Unicode
73124 6cfee8de0dea Refactor a bit the codecs doc. Ezio Melotti <ezio.melotti@gmail.com> parents: 72162 diff changeset	787 code point, is to store each codepoint as four consecutive bytes. There are two
6cfee8de0dea Refactor a bit the codecs doc. Ezio Melotti <ezio.melotti@gmail.com> parents: 72162 diff changeset	788 possibilities: store the bytes in big endian or in little endian order. These
6cfee8de0dea Refactor a bit the codecs doc. Ezio Melotti <ezio.melotti@gmail.com> parents: 72162 diff changeset	789 two encodings are called ``UTF-32-BE`` and ``UTF-32-LE`` respectively. Their
6cfee8de0dea Refactor a bit the codecs doc. Ezio Melotti <ezio.melotti@gmail.com> parents: 72162 diff changeset	790 disadvantage is that if e.g. you use ``UTF-32-BE`` on a little endian machine you
6cfee8de0dea Refactor a bit the codecs doc. Ezio Melotti <ezio.melotti@gmail.com> parents: 72162 diff changeset	791 will always have to swap bytes on encoding and decoding. ``UTF-32`` avoids this
6cfee8de0dea Refactor a bit the codecs doc. Ezio Melotti <ezio.melotti@gmail.com> parents: 72162 diff changeset	792 problem: bytes will always be in natural endianness. When these bytes are read
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	793 by a CPU with a different endianness, then bytes have to be swapped though. To
73124 6cfee8de0dea Refactor a bit the codecs doc. Ezio Melotti <ezio.melotti@gmail.com> parents: 72162 diff changeset	794 be able to detect the endianness of a ``UTF-16`` or ``UTF-32`` byte sequence,
6cfee8de0dea Refactor a bit the codecs doc. Ezio Melotti <ezio.melotti@gmail.com> parents: 72162 diff changeset	795 there's the so called BOM ("Byte Order Mark"). This is the Unicode character
6cfee8de0dea Refactor a bit the codecs doc. Ezio Melotti <ezio.melotti@gmail.com> parents: 72162 diff changeset	796 ``U+FEFF``. This character can be prepended to every ``UTF-16`` or ``UTF-32``
6cfee8de0dea Refactor a bit the codecs doc. Ezio Melotti <ezio.melotti@gmail.com> parents: 72162 diff changeset	797 byte sequence. The byte swapped version of this character (``0xFFFE``) is an
6cfee8de0dea Refactor a bit the codecs doc. Ezio Melotti <ezio.melotti@gmail.com> parents: 72162 diff changeset	798 illegal character that may not appear in a Unicode text. So when the
6cfee8de0dea Refactor a bit the codecs doc. Ezio Melotti <ezio.melotti@gmail.com> parents: 72162 diff changeset	799 first character in an ``UTF-16`` or ``UTF-32`` byte sequence
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	800 appears to be a ``U+FFFE`` the bytes have to be swapped on decoding.
73124 6cfee8de0dea Refactor a bit the codecs doc. Ezio Melotti <ezio.melotti@gmail.com> parents: 72162 diff changeset	801 Unfortunately the character ``U+FEFF`` had a second purpose as
6cfee8de0dea Refactor a bit the codecs doc. Ezio Melotti <ezio.melotti@gmail.com> parents: 72162 diff changeset	802 a ``ZERO WIDTH NO-BREAK SPACE``: a character that has no width and doesn't allow
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	803 a word to be split. It can e.g. be used to give hints to a ligature algorithm.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	804 With Unicode 4.0 using ``U+FEFF`` as a ``ZERO WIDTH NO-BREAK SPACE`` has been
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	805 deprecated (with ``U+2060`` (``WORD JOINER``) assuming this role). Nevertheless
73124 6cfee8de0dea Refactor a bit the codecs doc. Ezio Melotti <ezio.melotti@gmail.com> parents: 72162 diff changeset	806 Unicode software still must be able to handle ``U+FEFF`` in both roles: as a BOM
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	807 it's a device to determine the storage layout of the encoded bytes, and vanishes
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	808 once the byte sequence has been decoded into a Unicode string; as a ``ZERO WIDTH
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	809 NO-BREAK SPACE`` it's a normal character that will be decoded like any other.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	810
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	811 There's another encoding that is able to encoding the full range of Unicode
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	812 characters: UTF-8. UTF-8 is an 8-bit encoding, which means there are no issues
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	813 with byte order in UTF-8. Each byte in a UTF-8 byte sequence consists of two
73124 6cfee8de0dea Refactor a bit the codecs doc. Ezio Melotti <ezio.melotti@gmail.com> parents: 72162 diff changeset	814 parts: marker bits (the most significant bits) and payload bits. The marker bits
72162 4d584ebbfa77 From RFC 3629 5- and 6-bytes UTF-8 sequences are invalid, so remove them from the doc. Ezio Melotti <ezio.melotti@gmail.com> parents: 69331 diff changeset	815 are a sequence of zero to four ``1`` bits followed by a ``0`` bit. Unicode characters are
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	816 encoded like this (with x being payload bits, which when concatenated give the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	817 Unicode character):
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	818
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	819 +-----------------------------------+----------------------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	820 \| Range \| Encoding \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	821 +===================================+==============================================+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	822 \| ``U-00000000`` ... ``U-0000007F`` \| 0xxxxxxx \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	823 +-----------------------------------+----------------------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	824 \| ``U-00000080`` ... ``U-000007FF`` \| 110xxxxx 10xxxxxx \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	825 +-----------------------------------+----------------------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	826 \| ``U-00000800`` ... ``U-0000FFFF`` \| 1110xxxx 10xxxxxx 10xxxxxx \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	827 +-----------------------------------+----------------------------------------------+
72162 4d584ebbfa77 From RFC 3629 5- and 6-bytes UTF-8 sequences are invalid, so remove them from the doc. Ezio Melotti <ezio.melotti@gmail.com> parents: 69331 diff changeset	828 \| ``U-00010000`` ... ``U-0010FFFF`` \| 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx \|
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	829 +-----------------------------------+----------------------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	830
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	831 The least significant bit of the Unicode character is the rightmost x bit.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	832
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	833 As UTF-8 is an 8-bit encoding no BOM is required and any ``U+FEFF`` character in
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	834 the decoded Unicode string (even if it's the first character) is treated as a
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	835 ``ZERO WIDTH NO-BREAK SPACE``.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	836
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	837 Without external information it's impossible to reliably determine which
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	838 encoding was used for encoding a Unicode string. Each charmap encoding can
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	839 decode any random byte sequence. However that's not possible with UTF-8, as
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	840 UTF-8 byte sequences have a structure that doesn't allow arbitrary byte
43185 9b1833fc5c7b Fix typo. Walter Dörwald <walter@livinglogic.de> parents: 43184 diff changeset	841 sequences. To increase the reliability with which a UTF-8 encoding can be
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	842 detected, Microsoft invented a variant of UTF-8 (that Python 2.5 calls
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	843 ``"utf-8-sig"``) for its Notepad program: Before any of the Unicode characters
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	844 is written to the file, a UTF-8 encoded BOM (which looks like this as a byte
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	845 sequence: ``0xef``, ``0xbb``, ``0xbf``) is written. As it's rather improbable
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	846 that any charmap encoded file starts with these byte values (which would e.g.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	847 map to
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	848
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	849 \| LATIN SMALL LETTER I WITH DIAERESIS
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	850 \| RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	851 \| INVERTED QUESTION MARK
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	852
73124 6cfee8de0dea Refactor a bit the codecs doc. Ezio Melotti <ezio.melotti@gmail.com> parents: 72162 diff changeset	853 in iso-8859-1), this increases the probability that a ``utf-8-sig`` encoding can be
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	854 correctly guessed from the byte sequence. So here the BOM is not used to be able
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	855 to determine the byte order used for generating the byte sequence, but as a
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	856 signature that helps in guessing the encoding. On encoding the utf-8-sig codec
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	857 will write ``0xef``, ``0xbb``, ``0xbf`` as the first three bytes to the file. On
73124 6cfee8de0dea Refactor a bit the codecs doc. Ezio Melotti <ezio.melotti@gmail.com> parents: 72162 diff changeset	858 decoding ``utf-8-sig`` will skip those three bytes if they appear as the first
6cfee8de0dea Refactor a bit the codecs doc. Ezio Melotti <ezio.melotti@gmail.com> parents: 72162 diff changeset	859 three bytes in the file. In UTF-8, the use of the BOM is discouraged and
6cfee8de0dea Refactor a bit the codecs doc. Ezio Melotti <ezio.melotti@gmail.com> parents: 72162 diff changeset	860 should generally be avoided.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	861
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	862
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	863 .. _standard-encodings:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	864
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	865 Standard Encodings
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	866 ------------------
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	867
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	868 Python comes with a number of codecs built-in, either implemented as C functions
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	869 or with dictionaries as mapping tables. The following table lists the codecs by
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	870 name, together with a few common aliases, and the languages for which the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	871 encoding is likely used. Neither the list of aliases nor the list of languages
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	872 is meant to be exhaustive. Notice that spelling alternatives that only differ in
56820 85aa706f7126 Merged revisions 74492,74531,74545-74550,74553-74555,74588,74603,74608,74614,74616-74618,74631-74633,74652-74653,74666,74671,74737,74739,74779,74781-74782,74784,74791,74793,74818-74820,74822,74832 via svnmerge from Georg Brandl <georg@python.org> parents: 49677 diff changeset	873 case or use a hyphen instead of an underscore are also valid aliases; therefore,
85aa706f7126 Merged revisions 74492,74531,74545-74550,74553-74555,74588,74603,74608,74614,74616-74618,74631-74633,74652-74653,74666,74671,74737,74739,74779,74781-74782,74784,74791,74793,74818-74820,74822,74832 via svnmerge from Georg Brandl <georg@python.org> parents: 49677 diff changeset	874 e.g. ``'utf-8'`` is a valid alias for the ``'utf_8'`` codec.
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	875
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	876 Many of the character sets support the same languages. They vary in individual
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	877 characters (e.g. whether the EURO SIGN is supported or not), and in the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	878 assignment of characters to code positions. For the European languages in
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	879 particular, the following variants typically exist:
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	880
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	881 * an ISO 8859 codeset
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	882
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	883 * a Microsoft Windows code page, which is typically derived from a 8859 codeset,
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	884 but replaces control characters with additional graphic characters
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	885
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	886 * an IBM EBCDIC code page
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	887
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	888 * an IBM PC code page, which is ASCII compatible
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	889
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	890 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	891 \| Codec \| Aliases \| Languages \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	892 +=================+================================+================================+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	893 \| ascii \| 646, us-ascii \| English \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	894 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	895 \| big5 \| big5-tw, csbig5 \| Traditional Chinese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	896 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	897 \| big5hkscs \| big5-hkscs, hkscs \| Traditional Chinese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	898 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	899 \| cp037 \| IBM037, IBM039 \| English \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	900 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	901 \| cp424 \| EBCDIC-CP-HE, IBM424 \| Hebrew \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	902 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	903 \| cp437 \| 437, IBM437 \| English \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	904 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	905 \| cp500 \| EBCDIC-CP-BE, EBCDIC-CP-CH, \| Western Europe \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	906 \| \| IBM500 \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	907 +-----------------+--------------------------------+--------------------------------+
55252 0ed405c05cc3 Document the newly added codec Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 49677 diff changeset	908 \| cp720 \| \| Arabic \|
0ed405c05cc3 Document the newly added codec Amaury Forgeot d'Arc <amauryfa@gmail.com> parents: 49677 diff changeset	909 +-----------------+--------------------------------+--------------------------------+
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	910 \| cp737 \| \| Greek \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	911 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	912 \| cp775 \| IBM775 \| Baltic languages \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	913 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	914 \| cp850 \| 850, IBM850 \| Western Europe \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	915 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	916 \| cp852 \| 852, IBM852 \| Central and Eastern Europe \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	917 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	918 \| cp855 \| 855, IBM855 \| Bulgarian, Byelorussian, \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	919 \| \| \| Macedonian, Russian, Serbian \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	920 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	921 \| cp856 \| \| Hebrew \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	922 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	923 \| cp857 \| 857, IBM857 \| Turkish \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	924 +-----------------+--------------------------------+--------------------------------+
61884 c56fb63c648f #8016: add the CP858 codec (approved by Benjamin). (Also add CP720 to the tests, it was missing there.) Georg Brandl <georg@python.org> parents: 59611 diff changeset	925 \| cp858 \| 858, IBM858 \| Western Europe \|
c56fb63c648f #8016: add the CP858 codec (approved by Benjamin). (Also add CP720 to the tests, it was missing there.) Georg Brandl <georg@python.org> parents: 59611 diff changeset	926 +-----------------+--------------------------------+--------------------------------+
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	927 \| cp860 \| 860, IBM860 \| Portuguese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	928 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	929 \| cp861 \| 861, CP-IS, IBM861 \| Icelandic \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	930 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	931 \| cp862 \| 862, IBM862 \| Hebrew \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	932 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	933 \| cp863 \| 863, IBM863 \| Canadian \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	934 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	935 \| cp864 \| IBM864 \| Arabic \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	936 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	937 \| cp865 \| 865, IBM865 \| Danish, Norwegian \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	938 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	939 \| cp866 \| 866, IBM866 \| Russian \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	940 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	941 \| cp869 \| 869, CP-GR, IBM869 \| Greek \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	942 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	943 \| cp874 \| \| Thai \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	944 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	945 \| cp875 \| \| Greek \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	946 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	947 \| cp932 \| 932, ms932, mskanji, ms-kanji \| Japanese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	948 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	949 \| cp949 \| 949, ms949, uhc \| Korean \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	950 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	951 \| cp950 \| 950, ms950 \| Traditional Chinese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	952 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	953 \| cp1006 \| \| Urdu \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	954 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	955 \| cp1026 \| ibm1026 \| Turkish \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	956 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	957 \| cp1140 \| ibm1140 \| Western Europe \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	958 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	959 \| cp1250 \| windows-1250 \| Central and Eastern Europe \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	960 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	961 \| cp1251 \| windows-1251 \| Bulgarian, Byelorussian, \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	962 \| \| \| Macedonian, Russian, Serbian \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	963 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	964 \| cp1252 \| windows-1252 \| Western Europe \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	965 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	966 \| cp1253 \| windows-1253 \| Greek \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	967 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	968 \| cp1254 \| windows-1254 \| Turkish \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	969 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	970 \| cp1255 \| windows-1255 \| Hebrew \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	971 +-----------------+--------------------------------+--------------------------------+
56822 c4c429b08a57 Merged revisions 74861-74863,74876,74896,74930,74933,74952-74953,75015,75019,75260-75263,75265-75266,75289 via svnmerge from Georg Brandl <georg@python.org> parents: 56820 diff changeset	972 \| cp1256 \| windows-1256 \| Arabic \|
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	973 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	974 \| cp1257 \| windows-1257 \| Baltic languages \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	975 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	976 \| cp1258 \| windows-1258 \| Vietnamese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	977 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	978 \| euc_jp \| eucjp, ujis, u-jis \| Japanese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	979 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	980 \| euc_jis_2004 \| jisx0213, eucjis2004 \| Japanese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	981 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	982 \| euc_jisx0213 \| eucjisx0213 \| Japanese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	983 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	984 \| euc_kr \| euckr, korean, ksc5601, \| Korean \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	985 \| \| ks_c-5601, ks_c-5601-1987, \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	986 \| \| ksx1001, ks_x-1001 \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	987 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	988 \| gb2312 \| chinese, csiso58gb231280, euc- \| Simplified Chinese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	989 \| \| cn, euccn, eucgb2312-cn, \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	990 \| \| gb2312-1980, gb2312-80, iso- \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	991 \| \| ir-58 \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	992 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	993 \| gbk \| 936, cp936, ms936 \| Unified Chinese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	994 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	995 \| gb18030 \| gb18030-2000 \| Unified Chinese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	996 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	997 \| hz \| hzgb, hz-gb, hz-gb-2312 \| Simplified Chinese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	998 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	999 \| iso2022_jp \| csiso2022jp, iso2022jp, \| Japanese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1000 \| \| iso-2022-jp \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1001 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1002 \| iso2022_jp_1 \| iso2022jp-1, iso-2022-jp-1 \| Japanese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1003 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1004 \| iso2022_jp_2 \| iso2022jp-2, iso-2022-jp-2 \| Japanese, Korean, Simplified \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1005 \| \| \| Chinese, Western Europe, Greek \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1006 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1007 \| iso2022_jp_2004 \| iso2022jp-2004, \| Japanese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1008 \| \| iso-2022-jp-2004 \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1009 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1010 \| iso2022_jp_3 \| iso2022jp-3, iso-2022-jp-3 \| Japanese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1011 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1012 \| iso2022_jp_ext \| iso2022jp-ext, iso-2022-jp-ext \| Japanese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1013 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1014 \| iso2022_kr \| csiso2022kr, iso2022kr, \| Korean \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1015 \| \| iso-2022-kr \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1016 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1017 \| latin_1 \| iso-8859-1, iso8859-1, 8859, \| West Europe \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1018 \| \| cp819, latin, latin1, L1 \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1019 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1020 \| iso8859_2 \| iso-8859-2, latin2, L2 \| Central and Eastern Europe \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1021 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1022 \| iso8859_3 \| iso-8859-3, latin3, L3 \| Esperanto, Maltese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1023 +-----------------+--------------------------------+--------------------------------+
45429 853af5bbb497 A lot more typo fixes by Ori Avtalion. Georg Brandl <georg@python.org> parents: 45376 diff changeset	1024 \| iso8859_4 \| iso-8859-4, latin4, L4 \| Baltic languages \|
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1025 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1026 \| iso8859_5 \| iso-8859-5, cyrillic \| Bulgarian, Byelorussian, \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1027 \| \| \| Macedonian, Russian, Serbian \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1028 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1029 \| iso8859_6 \| iso-8859-6, arabic \| Arabic \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1030 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1031 \| iso8859_7 \| iso-8859-7, greek, greek8 \| Greek \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1032 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1033 \| iso8859_8 \| iso-8859-8, hebrew \| Hebrew \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1034 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1035 \| iso8859_9 \| iso-8859-9, latin5, L5 \| Turkish \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1036 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1037 \| iso8859_10 \| iso-8859-10, latin6, L6 \| Nordic languages \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1038 +-----------------+--------------------------------+--------------------------------+
59845 f96c8293dd7f Merged revisions 78859-78860,78952,79168-79169,79173,79176,79178-79179,79181,79184-79185,79192,79212 via svnmerge from Georg Brandl <georg@python.org> parents: 59176 diff changeset	1039 \| iso8859_13 \| iso-8859-13, latin7, L7 \| Baltic languages \|
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1040 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1041 \| iso8859_14 \| iso-8859-14, latin8, L8 \| Celtic languages \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1042 +-----------------+--------------------------------+--------------------------------+
59845 f96c8293dd7f Merged revisions 78859-78860,78952,79168-79169,79173,79176,79178-79179,79181,79184-79185,79192,79212 via svnmerge from Georg Brandl <georg@python.org> parents: 59176 diff changeset	1043 \| iso8859_15 \| iso-8859-15, latin9, L9 \| Western Europe \|
f96c8293dd7f Merged revisions 78859-78860,78952,79168-79169,79173,79176,79178-79179,79181,79184-79185,79192,79212 via svnmerge from Georg Brandl <georg@python.org> parents: 59176 diff changeset	1044 +-----------------+--------------------------------+--------------------------------+
f96c8293dd7f Merged revisions 78859-78860,78952,79168-79169,79173,79176,79178-79179,79181,79184-79185,79192,79212 via svnmerge from Georg Brandl <georg@python.org> parents: 59176 diff changeset	1045 \| iso8859_16 \| iso-8859-16, latin10, L10 \| South-Eastern Europe \|
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1046 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1047 \| johab \| cp1361, ms1361 \| Korean \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1048 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1049 \| koi8_r \| \| Russian \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1050 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1051 \| koi8_u \| \| Ukrainian \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1052 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1053 \| mac_cyrillic \| maccyrillic \| Bulgarian, Byelorussian, \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1054 \| \| \| Macedonian, Russian, Serbian \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1055 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1056 \| mac_greek \| macgreek \| Greek \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1057 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1058 \| mac_iceland \| maciceland \| Icelandic \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1059 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1060 \| mac_latin2 \| maclatin2, maccentraleurope \| Central and Eastern Europe \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1061 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1062 \| mac_roman \| macroman \| Western Europe \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1063 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1064 \| mac_turkish \| macturkish \| Turkish \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1065 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1066 \| ptcp154 \| csptcp154, pt154, cp154, \| Kazakh \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1067 \| \| cyrillic-asian \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1068 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1069 \| shift_jis \| csshiftjis, shiftjis, sjis, \| Japanese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1070 \| \| s_jis \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1071 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1072 \| shift_jis_2004 \| shiftjis2004, sjis_2004, \| Japanese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1073 \| \| sjis2004 \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1074 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1075 \| shift_jisx0213 \| shiftjisx0213, sjisx0213, \| Japanese \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1076 \| \| s_jisx0213 \| \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1077 +-----------------+--------------------------------+--------------------------------+
42649 c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 42603 diff changeset	1078 \| utf_32 \| U32, utf32 \| all languages \|
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 42603 diff changeset	1079 +-----------------+--------------------------------+--------------------------------+
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 42603 diff changeset	1080 \| utf_32_be \| UTF-32BE \| all languages \|
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 42603 diff changeset	1081 +-----------------+--------------------------------+--------------------------------+
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 42603 diff changeset	1082 \| utf_32_le \| UTF-32LE \| all languages \|
c916731321c0 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. Walter Dörwald <walter@livinglogic.de> parents: 42603 diff changeset	1083 +-----------------+--------------------------------+--------------------------------+
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1084 \| utf_16 \| U16, utf16 \| all languages \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1085 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1086 \| utf_16_be \| UTF-16BE \| all languages (BMP only) \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1087 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1088 \| utf_16_le \| UTF-16LE \| all languages (BMP only) \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1089 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1090 \| utf_7 \| U7, unicode-1-1-utf-7 \| all languages \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1091 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1092 \| utf_8 \| U8, UTF, utf8 \| all languages \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1093 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1094 \| utf_8_sig \| \| all languages \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1095 +-----------------+--------------------------------+--------------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1096
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1097 A number of codecs are specific to Python, so their codec names have no meaning
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1098 outside Python. Some of them don't convert from Unicode strings to byte strings,
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1099 but instead use the property of the Python codecs machinery that any bijective
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1100 function with one argument can be considered as an encoding.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1101
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1102 For the codecs listed below, the result in the "encoding" direction is always a
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1103 byte string. The result of the "decoding" direction is listed as operand type in
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1104 the table.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1105
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1106 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1107 \| Codec \| Aliases \| Operand type \| Purpose \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1108 +====================+===========================+================+===========================+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1109 \| base64_codec \| base64, base-64 \| byte string \| Convert operand to MIME \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1110 \| \| \| \| base64 \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1111 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1112 \| bz2_codec \| bz2 \| byte string \| Compress the operand \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1113 \| \| \| \| using bz2 \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1114 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1115 \| hex_codec \| hex \| byte string \| Convert operand to \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1116 \| \| \| \| hexadecimal \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1117 \| \| \| \| representation, with two \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1118 \| \| \| \| digits per byte \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1119 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1120 \| idna \| \| Unicode string \| Implements :rfc:`3490`, \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1121 \| \| \| \| see also \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1122 \| \| \| \| :mod:`encodings.idna` \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1123 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1124 \| mbcs \| dbcs \| Unicode string \| Windows only: Encode \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1125 \| \| \| \| operand according to the \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1126 \| \| \| \| ANSI codepage (CP_ACP) \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1127 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1128 \| palmos \| \| Unicode string \| Encoding of PalmOS 3.5 \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1129 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1130 \| punycode \| \| Unicode string \| Implements :rfc:`3492` \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1131 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1132 \| quopri_codec \| quopri, quoted-printable, \| byte string \| Convert operand to MIME \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1133 \| \| quotedprintable \| \| quoted printable \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1134 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1135 \| raw_unicode_escape \| \| Unicode string \| Produce a string that is \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1136 \| \| \| \| suitable as raw Unicode \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1137 \| \| \| \| literal in Python source \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1138 \| \| \| \| code \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1139 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1140 \| rot_13 \| rot13 \| Unicode string \| Returns the Caesar-cypher \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1141 \| \| \| \| encryption of the operand \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1142 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1143 \| string_escape \| \| byte string \| Produce a string that is \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1144 \| \| \| \| suitable as string \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1145 \| \| \| \| literal in Python source \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1146 \| \| \| \| code \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1147 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1148 \| undefined \| \| any \| Raise an exception for \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1149 \| \| \| \| all conversions. Can be \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1150 \| \| \| \| used as the system \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1151 \| \| \| \| encoding if no automatic \|
44103 1e266015cf5b Add more entries to the glossary. Georg Brandl <georg@python.org> parents: 43629 diff changeset	1152 \| \| \| \| :term:`coercion` between \|
1e266015cf5b Add more entries to the glossary. Georg Brandl <georg@python.org> parents: 43629 diff changeset	1153 \| \| \| \| byte and Unicode strings \|
1e266015cf5b Add more entries to the glossary. Georg Brandl <georg@python.org> parents: 43629 diff changeset	1154 \| \| \| \| is desired. \|
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1155 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1156 \| unicode_escape \| \| Unicode string \| Produce a string that is \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1157 \| \| \| \| suitable as Unicode \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1158 \| \| \| \| literal in Python source \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1159 \| \| \| \| code \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1160 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1161 \| unicode_internal \| \| Unicode string \| Return the internal \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1162 \| \| \| \| representation of the \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1163 \| \| \| \| operand \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1164 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1165 \| uu_codec \| uu \| byte string \| Convert the operand using \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1166 \| \| \| \| uuencode \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1167 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1168 \| zlib_codec \| zip, zlib \| byte string \| Compress the operand \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1169 \| \| \| \| using gzip \|
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1170 +--------------------+---------------------------+----------------+---------------------------+
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1171
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1172 .. versionadded:: 2.3
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1173 The ``idna`` and ``punycode`` encodings.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1174
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1175
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1176 :mod:`encodings.idna` --- Internationalized Domain Names in Applications
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1177 ------------------------------------------------------------------------
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1178
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1179 .. module:: encodings.idna
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1180 :synopsis: Internationalized Domain Names implementation
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1181 .. moduleauthor:: Martin v. Löwis
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1182
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1183 .. versionadded:: 2.3
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1184
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1185 This module implements :rfc:`3490` (Internationalized Domain Names in
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1186 Applications) and :rfc:`3492` (Nameprep: A Stringprep Profile for
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1187 Internationalized Domain Names (IDN)). It builds upon the ``punycode`` encoding
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1188 and :mod:`stringprep`.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1189
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1190 These RFCs together define a protocol to support non-ASCII characters in domain
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1191 names. A domain name containing non-ASCII characters (such as
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1192 ``www.Alliancefrançaise.nu``) is converted into an ASCII-compatible encoding
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1193 (ACE, such as ``www.xn--alliancefranaise-npb.nu``). The ACE form of the domain
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1194 name is then used in all places where arbitrary characters are not allowed by
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1195 the protocol, such as DNS queries, HTTP :mailheader:`Host` fields, and so
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1196 on. This conversion is carried out in the application; if possible invisible to
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1197 the user: The application should transparently convert Unicode domain labels to
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1198 IDNA on the wire, and convert back ACE labels to Unicode before presenting them
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1199 to the user.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1200
69331 00d6e594a40e Transplant: Fix wording and clarify that the IDNA codec operates on full domain names. R David Murray <rdmurray@bitdance.com> parents: 68256 diff changeset	1201 Python supports this conversion in several ways: the ``idna`` codec performs
00d6e594a40e Transplant: Fix wording and clarify that the IDNA codec operates on full domain names. R David Murray <rdmurray@bitdance.com> parents: 68256 diff changeset	1202 conversion between Unicode and ACE, separating an input string into labels
00d6e594a40e Transplant: Fix wording and clarify that the IDNA codec operates on full domain names. R David Murray <rdmurray@bitdance.com> parents: 68256 diff changeset	1203 based on the separator characters defined in `section 3.1`_ (1) of :rfc:`3490`
00d6e594a40e Transplant: Fix wording and clarify that the IDNA codec operates on full domain names. R David Murray <rdmurray@bitdance.com> parents: 68256 diff changeset	1204 and converting each label to ACE as required, and conversely separating an input
00d6e594a40e Transplant: Fix wording and clarify that the IDNA codec operates on full domain names. R David Murray <rdmurray@bitdance.com> parents: 68256 diff changeset	1205 byte string into labels based on the ``.`` separator and converting any ACE
00d6e594a40e Transplant: Fix wording and clarify that the IDNA codec operates on full domain names. R David Murray <rdmurray@bitdance.com> parents: 68256 diff changeset	1206 labels found into unicode. Furthermore, the :mod:`socket` module
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1207 transparently converts Unicode host names to ACE, so that applications need not
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1208 be concerned about converting host names themselves when they pass them to the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1209 socket module. On top of that, modules that have host names as function
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1210 parameters, such as :mod:`httplib` and :mod:`ftplib`, accept Unicode host names
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1211 (:mod:`httplib` then also transparently sends an IDNA hostname in the
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1212 :mailheader:`Host` field if it sends that field at all).
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1213
69331 00d6e594a40e Transplant: Fix wording and clarify that the IDNA codec operates on full domain names. R David Murray <rdmurray@bitdance.com> parents: 68256 diff changeset	1214 .. _section 3.1: http://tools.ietf.org/html/rfc3490#section-3.1
00d6e594a40e Transplant: Fix wording and clarify that the IDNA codec operates on full domain names. R David Murray <rdmurray@bitdance.com> parents: 68256 diff changeset	1215
42603 c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1216 When receiving host names from the wire (such as in reverse name lookup), no
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1217 automatic conversion to Unicode is performed: Applications wishing to present
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1218 such host names to the user should decode them to Unicode.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1219
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1220 The module :mod:`encodings.idna` also implements the nameprep procedure, which
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1221 performs certain normalizations on host names, to achieve case-insensitivity of
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1222 international domain names, and to unify similar characters. The nameprep
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1223 functions can be used directly if desired.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1224
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1225
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1226 .. function:: nameprep(label)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1227
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1228 Return the nameprepped version of label. The implementation currently assumes
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1229 query strings, so ``AllowUnassigned`` is true.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1230
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1231
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1232 .. function:: ToASCII(label)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1233
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1234 Convert a label to ASCII, as specified in :rfc:`3490`. ``UseSTD3ASCIIRules`` is
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1235 assumed to be false.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1236
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1237
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1238 .. function:: ToUnicode(label)
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1239
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1240 Convert a label to Unicode, as specified in :rfc:`3490`.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1241
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1242
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1243 :mod:`encodings.utf_8_sig` --- UTF-8 codec with BOM signature
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1244 -------------------------------------------------------------
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1245
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1246 .. module:: encodings.utf_8_sig
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1247 :synopsis: UTF-8 codec with BOM signature
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1248 .. moduleauthor:: Walter Dörwald
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1249
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1250 .. versionadded:: 2.5
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1251
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1252 This module implements a variant of the UTF-8 codec: On encoding a UTF-8 encoded
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1253 BOM will be prepended to the UTF-8 encoded bytes. For the stateful encoder this
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1254 is only done once (on the first write to the byte stream). For decoding an
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1255 optional UTF-8 encoded BOM at the start of the data will be skipped.
c62f6669ae22 Move the 2.6 reST doc tree in place. Georg Brandl <georg@python.org> parents: diff changeset	1256

Mercurial > cpython-lru_cache-weakref

annotate Doc/library/codecs.rst @ 79675:c0a2294de787 2.7