Mercurial > cpython-lru_cache-weakref

annotate Doc/lib/libcodecs.tex @ 13163:a09e8635534d legacy-trunk

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Don't try to guess the name of a .def file -- if one is supplied, use it, otherwise just generate an '/export:' option.
author Greg Ward <gward@python.net>
date Thu, 29 Jun 2000 23:09:20 +0000
parents 559b25a79032
children bb364d6f9f69
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
1 \section{\module{codecs} ---
12274
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
2 Codec registry and base classes}
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
3
12274
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
4 \declaremodule{standard}{codecs}
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
5 \modulesynopsis{Encode and decode data and streams.}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
6 \moduleauthor{Marc-Andre Lemburg}{mal@lemburg.com}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
7 \sectionauthor{Marc-Andre Lemburg}{mal@lemburg.com}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
8
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
9
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
10 \index{Unicode}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
11 \index{Codecs}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
12 \indexii{Codecs}{encode}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
13 \indexii{Codecs}{decode}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
14 \index{streams}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
15 \indexii{stackable}{streams}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
16
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
17
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
18 This module defines base classes for standard Python codecs (encoders
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
19 and decoders) and provides access to the internal Python codec
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
20 registry which manages the codec lookup process.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
21
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
22 It defines the following functions:
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
23
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
24 \begin{funcdesc}{register}{search_function}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
25 Register a codec search function. Search functions are expected to
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
26 take one argument, the encoding name in all lower case letters, and
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
27 return a tuple of functions \code{(\var{encoder}, \var{decoder}, \var{stream_reader},
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
28 \var{stream_writer})} taking the following arguments:
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
29
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
30 \var{encoder} and \var{decoder}: These must be functions or methods
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
31 which have the same interface as the .encode/.decode methods of
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
32 Codec instances (see Codec Interface). The functions/methods are
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
33 expected to work in a stateless mode.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
34
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
35 \var{stream_reader} and \var{stream_writer}: These have to be
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
36 factory functions providing the following interface:
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
37
12274
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
38 \code{factory(\var{stream}, \var{errors}='strict')}
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
39
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
40 The factory functions must return objects providing the interfaces
12274
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
41 defined by the base classes \class{StreamWriter} and
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
42 \class{StreamReader}, respectively. Stream codecs can maintain
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
43 state.
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
44
12274
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
45 Possible values for errors are \code{'strict'} (raise an exception
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
46 in case of an encoding error), \code{'replace'} (replace malformed
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
47 data with a suitable replacement marker, such as \character{?}) and
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
48 \code{'ignore'} (ignore malformed data and continue without further
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
49 notice).
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
50
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
51 In case a search function cannot find a given encoding, it should
12274
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
52 return \code{None}.
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
53 \end{funcdesc}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
54
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
55 \begin{funcdesc}{lookup}{encoding}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
56 Looks up a codec tuple in the Python codec registry and returns the
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
57 function tuple as defined above.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
58
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
59 Encodings are first looked up in the registry's cache. If not found,
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
60 the list of registered search functions is scanned. If no codecs tuple
12274
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
61 is found, a \exception{LookupError} is raised. Otherwise, the codecs
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
62 tuple is stored in the cache and returned to the caller.
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
63 \end{funcdesc}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
64
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
65 To simplify working with encoded files or stream, the module
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
66 also defines these utility functions:
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
67
12274
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
68 \begin{funcdesc}{open}{filename, mode\optional{, encoding=None\optional{, errors='strict'\optional{, buffering=1}}}}
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
69 Open an encoded file using the given \var{mode} and return
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
70 a wrapped version providing transparent encoding/decoding.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
71
12274
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
72 \strong{Note:} The wrapped version will only accept the object format
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
73 defined by the codecs, i.e. Unicode objects for most builtin
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
74 codecs. Output is also codec dependent and will usually by Unicode as
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
75 well.
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
76
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
77 \var{encoding} specifies the encoding which is to be used for the
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
78 the file.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
79
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
80 \var{errors} may be given to define the error handling. It defaults
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
81 to 'strict' which causes a \exception{ValueError} to be raised in case
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
82 an encoding error occurs.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
83
12274
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
84 \var{buffering} has the same meaning as for the built-in
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
85 \function{open()} function. It defaults to line buffered.
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
86 \end{funcdesc}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
87
12274
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
88 \begin{funcdesc}{EncodedFile}{file, input\optional{, output=None\optional{, errors='strict'}}}
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
89
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
90 Return a wrapped version of file which provides transparent
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
91 encoding translation.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
92
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
93 Strings written to the wrapped file are interpreted according to the
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
94 given \var{input} encoding and then written to the original file as
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
95 string using the \var{output} encoding. The intermediate encoding will
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
96 usually be Unicode but depends on the specified codecs.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
97
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
98 If \var{output} is not given, it defaults to input.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
99
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
100 \var{errors} may be given to define the error handling. It defaults to
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
101 'strict' which causes \exception{ValueError} to be raised in case
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
102 an encoding error occurs.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
103 \end{funcdesc}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
104
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
105
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
106
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
107 ...XXX document codec base classes...
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
108
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
109
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
110
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
111 The module also provides the following constants which are useful
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
112 for reading and writing to platform dependent files:
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
113
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
114 \begin{datadesc}{BOM}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
115 \dataline{BOM_BE}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
116 \dataline{BOM_LE}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
117 \dataline{BOM32_BE}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
118 \dataline{BOM32_LE}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
119 \dataline{BOM64_BE}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
120 \dataline{BOM64_LE}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
121 These constants define the byte order marks (BOM) used in data
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
122 streams to indicate the byte order used in the stream or file.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
123 \constant{BOM} is either \constant{BOM_BE} or \constant{BOM_LE}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
124 depending on the platform's native byte order, while the others
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
125 represent big endian (\samp{_BE} suffix) and little endian
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
126 (\samp{_LE} suffix) byte order using 32-bit and 64-bit encodings.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
127 \end{datadesc}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
128