Mercurial > cpython-lru_cache-weakref

annotate Doc/lib/libcodecs.tex @ 14984:ece6c19edcc1 legacy-trunk

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Remove some debugging prints.
author Greg Ward <gward@python.net>
date Tue, 03 Oct 2000 03:32:37 +0000
parents bb364d6f9f69
children b2892cbc63db
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
1 \section{\module{codecs} ---
12274
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
2 Codec registry and base classes}
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
3
12274
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
4 \declaremodule{standard}{codecs}
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
5 \modulesynopsis{Encode and decode data and streams.}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
6 \moduleauthor{Marc-Andre Lemburg}{mal@lemburg.com}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
7 \sectionauthor{Marc-Andre Lemburg}{mal@lemburg.com}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
8
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
9
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
10 \index{Unicode}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
11 \index{Codecs}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
12 \indexii{Codecs}{encode}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
13 \indexii{Codecs}{decode}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
14 \index{streams}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
15 \indexii{stackable}{streams}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
16
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
17
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
18 This module defines base classes for standard Python codecs (encoders
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
19 and decoders) and provides access to the internal Python codec
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
20 registry which manages the codec lookup process.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
21
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
22 It defines the following functions:
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
23
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
24 \begin{funcdesc}{register}{search_function}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
25 Register a codec search function. Search functions are expected to
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
26 take one argument, the encoding name in all lower case letters, and
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
27 return a tuple of functions \code{(\var{encoder}, \var{decoder}, \var{stream_reader},
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
28 \var{stream_writer})} taking the following arguments:
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
29
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
30 \var{encoder} and \var{decoder}: These must be functions or methods
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
31 which have the same interface as the .encode/.decode methods of
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
32 Codec instances (see Codec Interface). The functions/methods are
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
33 expected to work in a stateless mode.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
34
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
35 \var{stream_reader} and \var{stream_writer}: These have to be
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
36 factory functions providing the following interface:
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
37
12274
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
38 \code{factory(\var{stream}, \var{errors}='strict')}
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
39
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
40 The factory functions must return objects providing the interfaces
12274
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
41 defined by the base classes \class{StreamWriter} and
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
42 \class{StreamReader}, respectively. Stream codecs can maintain
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
43 state.
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
44
12274
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
45 Possible values for errors are \code{'strict'} (raise an exception
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
46 in case of an encoding error), \code{'replace'} (replace malformed
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
47 data with a suitable replacement marker, such as \character{?}) and
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
48 \code{'ignore'} (ignore malformed data and continue without further
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
49 notice).
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
50
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
51 In case a search function cannot find a given encoding, it should
12274
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
52 return \code{None}.
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
53 \end{funcdesc}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
54
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
55 \begin{funcdesc}{lookup}{encoding}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
56 Looks up a codec tuple in the Python codec registry and returns the
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
57 function tuple as defined above.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
58
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
59 Encodings are first looked up in the registry's cache. If not found,
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
60 the list of registered search functions is scanned. If no codecs tuple
12274
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
61 is found, a \exception{LookupError} is raised. Otherwise, the codecs
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
62 tuple is stored in the cache and returned to the caller.
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
63 \end{funcdesc}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
64
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
65 To simplify working with encoded files or stream, the module
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
66 also defines these utility functions:
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
67
13696
bb364d6f9f69 Fix small typos and markup consistency nits.
Fred Drake <fdrake@acm.org>
parents: 12274
diff changeset
68 \begin{funcdesc}{open}{filename, mode\optional{, encoding\optional{,
bb364d6f9f69 Fix small typos and markup consistency nits.
Fred Drake <fdrake@acm.org>
parents: 12274
diff changeset
69 errors\optional{, buffering}}}}
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
70 Open an encoded file using the given \var{mode} and return
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
71 a wrapped version providing transparent encoding/decoding.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
72
12274
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
73 \strong{Note:} The wrapped version will only accept the object format
13696
bb364d6f9f69 Fix small typos and markup consistency nits.
Fred Drake <fdrake@acm.org>
parents: 12274
diff changeset
74 defined by the codecs, i.e.\ Unicode objects for most built-in
bb364d6f9f69 Fix small typos and markup consistency nits.
Fred Drake <fdrake@acm.org>
parents: 12274
diff changeset
75 codecs. Output is also codec-dependent and will usually be Unicode as
12274
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
76 well.
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
77
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
78 \var{encoding} specifies the encoding which is to be used for the
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
79 the file.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
80
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
81 \var{errors} may be given to define the error handling. It defaults
13696
bb364d6f9f69 Fix small typos and markup consistency nits.
Fred Drake <fdrake@acm.org>
parents: 12274
diff changeset
82 to \code{'strict'} which causes a \exception{ValueError} to be raised
bb364d6f9f69 Fix small typos and markup consistency nits.
Fred Drake <fdrake@acm.org>
parents: 12274
diff changeset
83 in case an encoding error occurs.
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
84
12274
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
85 \var{buffering} has the same meaning as for the built-in
559b25a79032 Make sure the \declaremodule uses the right name for the module!
Fred Drake <fdrake@acm.org>
parents: 12260
diff changeset
86 \function{open()} function. It defaults to line buffered.
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
87 \end{funcdesc}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
88
13696
bb364d6f9f69 Fix small typos and markup consistency nits.
Fred Drake <fdrake@acm.org>
parents: 12274
diff changeset
89 \begin{funcdesc}{EncodedFile}{file, input\optional{,
bb364d6f9f69 Fix small typos and markup consistency nits.
Fred Drake <fdrake@acm.org>
parents: 12274
diff changeset
90 output\optional{, errors}}}
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
91 Return a wrapped version of file which provides transparent
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
92 encoding translation.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
93
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
94 Strings written to the wrapped file are interpreted according to the
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
95 given \var{input} encoding and then written to the original file as
13696
bb364d6f9f69 Fix small typos and markup consistency nits.
Fred Drake <fdrake@acm.org>
parents: 12274
diff changeset
96 strings using the \var{output} encoding. The intermediate encoding will
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
97 usually be Unicode but depends on the specified codecs.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
98
13696
bb364d6f9f69 Fix small typos and markup consistency nits.
Fred Drake <fdrake@acm.org>
parents: 12274
diff changeset
99 If \var{output} is not given, it defaults to \var{input}.
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
100
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
101 \var{errors} may be given to define the error handling. It defaults to
13696
bb364d6f9f69 Fix small typos and markup consistency nits.
Fred Drake <fdrake@acm.org>
parents: 12274
diff changeset
102 \code{'strict'}, which causes \exception{ValueError} to be raised in case
12260
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
103 an encoding error occurs.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
104 \end{funcdesc}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
105
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
106
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
107
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
108 ...XXX document codec base classes...
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
109
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
110
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
111
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
112 The module also provides the following constants which are useful
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
113 for reading and writing to platform dependent files:
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
114
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
115 \begin{datadesc}{BOM}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
116 \dataline{BOM_BE}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
117 \dataline{BOM_LE}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
118 \dataline{BOM32_BE}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
119 \dataline{BOM32_LE}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
120 \dataline{BOM64_BE}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
121 \dataline{BOM64_LE}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
122 These constants define the byte order marks (BOM) used in data
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
123 streams to indicate the byte order used in the stream or file.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
124 \constant{BOM} is either \constant{BOM_BE} or \constant{BOM_LE}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
125 depending on the platform's native byte order, while the others
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
126 represent big endian (\samp{_BE} suffix) and little endian
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
127 (\samp{_LE} suffix) byte order using 32-bit and 64-bit encodings.
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
128 \end{datadesc}
8c0f863ba50c Marc-Andre Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents:
diff changeset
129