Mercurial > cpython-lru_cache-weakref

annotate Python/codecs.c @ 87281:116bd550e309

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Issue #9566, #19617: Fix more compiler warnings in compile.c on Windows 64-bit
author Victor Stinner <victor.stinner@gmail.com>
date Tue, 19 Nov 2013 23:56:34 +0100
parents 0d9624f2ff43
children d68df99d7a57
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
1 /* ------------------------------------------------------------------------
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
2
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
3 Python Codec Registry and support functions
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
4
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
5 Written by Marc-Andre Lemburg (mal@lemburg.com).
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
6
13828
254f8b7c4a83 Changing the CNRI copyright notice according to CNRI's instructions.
Guido van Rossum <guido@python.org>
parents: 13658
diff changeset
7 Copyright (c) Corporation for National Research Initiatives.
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
8
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
9 ------------------------------------------------------------------------ */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
10
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
11 #include "Python.h"
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
12 #include <ctype.h>
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
13
72918
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
14 const char *Py_hexdigits = "0123456789abcdef";
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
15
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
16 /* --- Codec Registry ----------------------------------------------------- */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
17
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
18 /* Import the standard encodings package which will register the first
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
19 codec search function.
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
20
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
21 This is done in a lazy way so that the Unicode implementation does
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
22 not downgrade startup time of scripts not needing it.
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
23
12195
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
24 ImportErrors are silently ignored by this function. Only one try is
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
25 made.
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
26
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
27 */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
28
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
29 static int _PyCodecRegistry_Init(void); /* Forward */
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
30
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
31 int PyCodec_Register(PyObject *search_function)
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
32 {
31797
846b3ee7e106 Changed random calls to PyThreadState_Get() to use the macro
Nicholas Bastin <nick.bastin@gmail.com>
parents: 31273
diff changeset
33 PyInterpreterState *interp = PyThreadState_GET()->interp;
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
34 if (interp->codec_search_path == NULL && _PyCodecRegistry_Init())
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
35 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
36 if (search_function == NULL) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
37 PyErr_BadArgument();
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
38 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
39 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
40 if (!PyCallable_Check(search_function)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
41 PyErr_SetString(PyExc_TypeError, "argument must be callable");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
42 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
43 }
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
44 return PyList_Append(interp->codec_search_path, search_function);
12195
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
45
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
46 onError:
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
47 return -1;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
48 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
49
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
50 /* Convert a string to a normalized Python string: all characters are
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
51 converted to lower case, spaces are replaced with underscores. */
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
52
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
53 static
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
54 PyObject *normalizestring(const char *string)
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
55 {
85157
e7f6cef7a4cc Issue #18722: Remove uses of the "register" keyword in C code.
Antoine Pitrou <solipsis@pitrou.net>
parents: 84571
diff changeset
56 size_t i;
13096
8c8de5b736dc Trent Mick's Win64 changes: size_t vs. int or long; also some overflow
Guido van Rossum <guido@python.org>
parents: 12648
diff changeset
57 size_t len = strlen(string);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
58 char *p;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
59 PyObject *v;
43603
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
60
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
61 if (len > PY_SSIZE_T_MAX) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
62 PyErr_SetString(PyExc_OverflowError, "string is too large");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
63 return NULL;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
64 }
43603
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
65
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
66 p = PyMem_Malloc(len + 1);
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
67 if (p == NULL)
84571
4975bcd67aa7 Issue #18408: normalizestring() now raises MemoryError on memory allocation failure
Victor Stinner <victor.stinner@gmail.com>
parents: 81031
diff changeset
68 return PyErr_NoMemory();
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
69 for (i = 0; i < len; i++) {
85157
e7f6cef7a4cc Issue #18722: Remove uses of the "register" keyword in C code.
Antoine Pitrou <solipsis@pitrou.net>
parents: 84571
diff changeset
70 char ch = string[i];
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
71 if (ch == ' ')
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
72 ch = '-';
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
73 else
71489
92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales.
Antoine Pitrou <solipsis@pitrou.net>
parents: 64590
diff changeset
74 ch = Py_TOLOWER(Py_CHARMASK(ch));
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
75 p[i] = ch;
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
76 }
43603
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
77 p[i] = '\0';
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
78 v = PyUnicode_FromString(p);
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
79 if (v == NULL)
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
80 return NULL;
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
81 PyMem_Free(p);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
82 return v;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
83 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
84
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
85 /* Lookup the given encoding and return a tuple providing the codec
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
86 facilities.
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
87
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
88 The encoding string is looked up converted to all lower-case
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
89 characters. This makes encodings looked up through this mechanism
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
90 effectively case-insensitive.
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
91
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
92 If no codec is found, a LookupError is set and NULL returned.
12195
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
93
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
94 As side effect, this tries to load the encodings package, if not
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
95 yet done. This is part of the lazy load strategy for the encodings
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
96 package.
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
97
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
98 */
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
99
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
100 PyObject *_PyCodec_Lookup(const char *encoding)
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
101 {
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
102 PyInterpreterState *interp;
12048
f87b56eebacc Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12025
diff changeset
103 PyObject *result, *args = NULL, *v;
38453
29ae5e0fc348 Much-needed merge (using svnmerge.py this time) of trunk changes into p3yk.
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
104 Py_ssize_t i, len;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
105
12648
2a53f5dc0590 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12248
diff changeset
106 if (encoding == NULL) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
107 PyErr_BadArgument();
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
108 goto onError;
12648
2a53f5dc0590 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12248
diff changeset
109 }
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
110
31797
846b3ee7e106 Changed random calls to PyThreadState_Get() to use the macro
Nicholas Bastin <nick.bastin@gmail.com>
parents: 31273
diff changeset
111 interp = PyThreadState_GET()->interp;
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
112 if (interp->codec_search_path == NULL && _PyCodecRegistry_Init())
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
113 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
114
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
115 /* Convert the encoding to a normalized Python string: all
13620
5b727ab37092 Spelling fixes supplied by Rob W. W. Hooft. All these are fixes in either
Thomas Wouters <thomas@python.org>
parents: 13330
diff changeset
116 characters are converted to lower case, spaces and hyphens are
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
117 replaced with underscores. */
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
118 v = normalizestring(encoding);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
119 if (v == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
120 goto onError;
43603
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
121 PyUnicode_InternInPlace(&v);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
122
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
123 /* First, try to lookup the name in the registry dictionary */
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
124 result = PyDict_GetItem(interp->codec_search_cache, v);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
125 if (result != NULL) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
126 Py_INCREF(result);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
127 Py_DECREF(v);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
128 return result;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
129 }
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
130
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
131 /* Next, scan the search functions in order of registration */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
132 args = PyTuple_New(1);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
133 if (args == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
134 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
135 PyTuple_SET_ITEM(args,0,v);
12048
f87b56eebacc Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12025
diff changeset
136
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
137 len = PyList_Size(interp->codec_search_path);
12048
f87b56eebacc Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12025
diff changeset
138 if (len < 0)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
139 goto onError;
12195
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
140 if (len == 0) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
141 PyErr_SetString(PyExc_LookupError,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
142 "no codec search functions registered: "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
143 "can't find encoding");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
144 goto onError;
12195
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
145 }
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
146
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
147 for (i = 0; i < len; i++) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
148 PyObject *func;
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
149
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
150 func = PyList_GetItem(interp->codec_search_path, i);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
151 if (func == NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
152 goto onError;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
153 result = PyEval_CallObject(func, args);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
154 if (result == NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
155 goto onError;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
156 if (result == Py_None) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
157 Py_DECREF(result);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
158 continue;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
159 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
160 if (!PyTuple_Check(result) || PyTuple_GET_SIZE(result) != 4) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
161 PyErr_SetString(PyExc_TypeError,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
162 "codec search functions must return 4-tuples");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
163 Py_DECREF(result);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
164 goto onError;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
165 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
166 break;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
167 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
168 if (i == len) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
169 /* XXX Perhaps we should cache misses too ? */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
170 PyErr_Format(PyExc_LookupError,
25441
d32588c88633 Add encoding name in LookupError. Fixes #615013. Will backport to 2.2.
Martin v. Löwis <martin@v.loewis.de>
parents: 25095
diff changeset
171 "unknown encoding: %s", encoding);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
172 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
173 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
174
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
175 /* Cache and return the result */
42534
5f9e79d1e158 Handle error
Neal Norwitz <nnorwitz@gmail.com>
parents: 42057
diff changeset
176 if (PyDict_SetItem(interp->codec_search_cache, v, result) < 0) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
177 Py_DECREF(result);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
178 goto onError;
42534
5f9e79d1e158 Handle error
Neal Norwitz <nnorwitz@gmail.com>
parents: 42057
diff changeset
179 }
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
180 Py_DECREF(args);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
181 return result;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
182
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
183 onError:
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
184 Py_XDECREF(args);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
185 return NULL;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
186 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
187
47496
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
188 /* Codec registry encoding check API. */
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
189
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
190 int PyCodec_KnownEncoding(const char *encoding)
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
191 {
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
192 PyObject *codecs;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
193
47496
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
194 codecs = _PyCodec_Lookup(encoding);
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
195 if (!codecs) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
196 PyErr_Clear();
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
197 return 0;
47496
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
198 }
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
199 else {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
200 Py_DECREF(codecs);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
201 return 1;
47496
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
202 }
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
203 }
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
204
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
205 static
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
206 PyObject *args_tuple(PyObject *object,
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
207 const char *errors)
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
208 {
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
209 PyObject *args;
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
210
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
211 args = PyTuple_New(1 + (errors != NULL));
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
212 if (args == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
213 return NULL;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
214 Py_INCREF(object);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
215 PyTuple_SET_ITEM(args,0,object);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
216 if (errors) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
217 PyObject *v;
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
218
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
219 v = PyUnicode_FromString(errors);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
220 if (v == NULL) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
221 Py_DECREF(args);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
222 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
223 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
224 PyTuple_SET_ITEM(args, 1, v);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
225 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
226 return args;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
227 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
228
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
229 /* Helper function to get a codec item */
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
230
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
231 static
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
232 PyObject *codec_getitem(const char *encoding, int index)
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
233 {
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
234 PyObject *codecs;
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
235 PyObject *v;
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
236
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
237 codecs = _PyCodec_Lookup(encoding);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
238 if (codecs == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
239 return NULL;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
240 v = PyTuple_GET_ITEM(codecs, index);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
241 Py_DECREF(codecs);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
242 Py_INCREF(v);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
243 return v;
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
244 }
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
245
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
246 /* Helper function to create an incremental codec. */
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
247
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
248 static
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
249 PyObject *codec_getincrementalcodec(const char *encoding,
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
250 const char *errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
251 const char *attrname)
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
252 {
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
253 PyObject *codecs, *ret, *inccodec;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
254
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
255 codecs = _PyCodec_Lookup(encoding);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
256 if (codecs == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
257 return NULL;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
258 inccodec = PyObject_GetAttrString(codecs, attrname);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
259 Py_DECREF(codecs);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
260 if (inccodec == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
261 return NULL;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
262 if (errors)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
263 ret = PyObject_CallFunction(inccodec, "s", errors);
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
264 else
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
265 ret = PyObject_CallFunction(inccodec, NULL);
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
266 Py_DECREF(inccodec);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
267 return ret;
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
268 }
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
269
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
270 /* Helper function to create a stream codec. */
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
271
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
272 static
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
273 PyObject *codec_getstreamcodec(const char *encoding,
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
274 PyObject *stream,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
275 const char *errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
276 const int index)
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
277 {
39442
654c380cf8b9 Merged revisions 46753-51188 via svnmerge from
Thomas Wouters <thomas@python.org>
parents: 38453
diff changeset
278 PyObject *codecs, *streamcodec, *codeccls;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
279
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
280 codecs = _PyCodec_Lookup(encoding);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
281 if (codecs == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
282 return NULL;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
283
39442
654c380cf8b9 Merged revisions 46753-51188 via svnmerge from
Thomas Wouters <thomas@python.org>
parents: 38453
diff changeset
284 codeccls = PyTuple_GET_ITEM(codecs, index);
654c380cf8b9 Merged revisions 46753-51188 via svnmerge from
Thomas Wouters <thomas@python.org>
parents: 38453
diff changeset
285 if (errors != NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
286 streamcodec = PyObject_CallFunction(codeccls, "Os", stream, errors);
39442
654c380cf8b9 Merged revisions 46753-51188 via svnmerge from
Thomas Wouters <thomas@python.org>
parents: 38453
diff changeset
287 else
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
288 streamcodec = PyObject_CallFunction(codeccls, "O", stream);
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
289 Py_DECREF(codecs);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
290 return streamcodec;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
291 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
292
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
293 /* Convenience APIs to query the Codec registry.
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
294
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
295 All APIs return a codec object with incremented refcount.
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
296
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
297 */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
298
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
299 PyObject *PyCodec_Encoder(const char *encoding)
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
300 {
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
301 return codec_getitem(encoding, 0);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
302 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
303
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
304 PyObject *PyCodec_Decoder(const char *encoding)
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
305 {
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
306 return codec_getitem(encoding, 1);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
307 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
308
36996
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 36751
diff changeset
309 PyObject *PyCodec_IncrementalEncoder(const char *encoding,
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
310 const char *errors)
36996
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 36751
diff changeset
311 {
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
312 return codec_getincrementalcodec(encoding, errors, "incrementalencoder");
36996
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 36751
diff changeset
313 }
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 36751
diff changeset
314
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 36751
diff changeset
315 PyObject *PyCodec_IncrementalDecoder(const char *encoding,
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
316 const char *errors)
36996
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 36751
diff changeset
317 {
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
318 return codec_getincrementalcodec(encoding, errors, "incrementaldecoder");
36996
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 36751
diff changeset
319 }
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 36751
diff changeset
320
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
321 PyObject *PyCodec_StreamReader(const char *encoding,
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
322 PyObject *stream,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
323 const char *errors)
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
324 {
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
325 return codec_getstreamcodec(encoding, stream, errors, 2);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
326 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
327
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
328 PyObject *PyCodec_StreamWriter(const char *encoding,
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
329 PyObject *stream,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
330 const char *errors)
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
331 {
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
332 return codec_getstreamcodec(encoding, stream, errors, 3);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
333 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
334
87084
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
335 /* Helper that tries to ensure the reported exception chain indicates the
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
336 * codec that was invoked to trigger the failure without changing the type
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
337 * of the exception raised.
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
338 */
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
339 static void
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
340 wrap_codec_error(const char *operation,
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
341 const char *encoding)
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
342 {
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
343 /* TrySetFromCause will replace the active exception with a suitably
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
344 * updated clone if it can, otherwise it will leave the original
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
345 * exception alone.
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
346 */
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
347 _PyErr_TrySetFromCause("%s with '%s' codec failed",
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
348 operation, encoding);
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
349 }
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
350
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
351 /* Encode an object (e.g. an Unicode object) using the given encoding
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
352 and return the resulting encoded object (usually a Python string).
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
353
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
354 errors is passed to the encoder factory as argument if non-NULL. */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
355
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
356 PyObject *PyCodec_Encode(PyObject *object,
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
357 const char *encoding,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
358 const char *errors)
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
359 {
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
360 PyObject *encoder = NULL;
35939
79374eb6db51 Fix a few more ref leaks. Backport candidate
Neal Norwitz <nnorwitz@gmail.com>
parents: 32533
diff changeset
361 PyObject *args = NULL, *result = NULL;
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
362 PyObject *v = NULL;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
363
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
364 encoder = PyCodec_Encoder(encoding);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
365 if (encoder == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
366 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
367
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
368 args = args_tuple(object, errors);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
369 if (args == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
370 goto onError;
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
371
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
372 result = PyEval_CallObject(encoder, args);
87109
4ea622c085ca Close 19609: narrow scope of codec exc chaining
Nick Coghlan <ncoghlan@gmail.com>
parents: 87084
diff changeset
373 if (result == NULL) {
4ea622c085ca Close 19609: narrow scope of codec exc chaining
Nick Coghlan <ncoghlan@gmail.com>
parents: 87084
diff changeset
374 wrap_codec_error("encoding", encoding);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
375 goto onError;
87109
4ea622c085ca Close 19609: narrow scope of codec exc chaining
Nick Coghlan <ncoghlan@gmail.com>
parents: 87084
diff changeset
376 }
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
377
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
378 if (!PyTuple_Check(result) ||
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
379 PyTuple_GET_SIZE(result) != 2) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
380 PyErr_SetString(PyExc_TypeError,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
381 "encoder must return a tuple (object, integer)");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
382 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
383 }
47496
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
384 v = PyTuple_GET_ITEM(result,0);
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
385 Py_INCREF(v);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
386 /* We don't check or use the second (integer) entry. */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
387
47496
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
388 Py_DECREF(args);
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
389 Py_DECREF(encoder);
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
390 Py_DECREF(result);
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
391 return v;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
392
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
393 onError:
35939
79374eb6db51 Fix a few more ref leaks. Backport candidate
Neal Norwitz <nnorwitz@gmail.com>
parents: 32533
diff changeset
394 Py_XDECREF(result);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
395 Py_XDECREF(args);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
396 Py_XDECREF(encoder);
47496
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
397 return NULL;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
398 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
399
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
400 /* Decode an object (usually a Python string) using the given encoding
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
401 and return an equivalent object (e.g. an Unicode object).
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
402
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
403 errors is passed to the decoder factory as argument if non-NULL. */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
404
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
405 PyObject *PyCodec_Decode(PyObject *object,
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
406 const char *encoding,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
407 const char *errors)
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
408 {
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
409 PyObject *decoder = NULL;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
410 PyObject *args = NULL, *result = NULL;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
411 PyObject *v;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
412
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
413 decoder = PyCodec_Decoder(encoding);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
414 if (decoder == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
415 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
416
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
417 args = args_tuple(object, errors);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
418 if (args == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
419 goto onError;
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
420
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
421 result = PyEval_CallObject(decoder,args);
87109
4ea622c085ca Close 19609: narrow scope of codec exc chaining
Nick Coghlan <ncoghlan@gmail.com>
parents: 87084
diff changeset
422 if (result == NULL) {
4ea622c085ca Close 19609: narrow scope of codec exc chaining
Nick Coghlan <ncoghlan@gmail.com>
parents: 87084
diff changeset
423 wrap_codec_error("decoding", encoding);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
424 goto onError;
87109
4ea622c085ca Close 19609: narrow scope of codec exc chaining
Nick Coghlan <ncoghlan@gmail.com>
parents: 87084
diff changeset
425 }
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
426 if (!PyTuple_Check(result) ||
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
427 PyTuple_GET_SIZE(result) != 2) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
428 PyErr_SetString(PyExc_TypeError,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
429 "decoder must return a tuple (object,integer)");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
430 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
431 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
432 v = PyTuple_GET_ITEM(result,0);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
433 Py_INCREF(v);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
434 /* We don't check or use the second (integer) entry. */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
435
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
436 Py_DECREF(args);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
437 Py_DECREF(decoder);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
438 Py_DECREF(result);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
439 return v;
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
440
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
441 onError:
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
442 Py_XDECREF(args);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
443 Py_XDECREF(decoder);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
444 Py_XDECREF(result);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
445 return NULL;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
446 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
447
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
448 /* Register the error handling callback function error under the name
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
449 name. This function will be called by the codec when it encounters
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
450 an unencodable characters/undecodable bytes and doesn't know the
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
451 callback name, when name is specified as the error parameter
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
452 in the call to the encode/decode function.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
453 Return 0 on success, -1 on error */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
454 int PyCodec_RegisterError(const char *name, PyObject *error)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
455 {
31797
846b3ee7e106 Changed random calls to PyThreadState_Get() to use the macro
Nicholas Bastin <nick.bastin@gmail.com>
parents: 31273
diff changeset
456 PyInterpreterState *interp = PyThreadState_GET()->interp;
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
457 if (interp->codec_search_path == NULL && _PyCodecRegistry_Init())
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
458 return -1;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
459 if (!PyCallable_Check(error)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
460 PyErr_SetString(PyExc_TypeError, "handler must be callable");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
461 return -1;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
462 }
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
463 return PyDict_SetItemString(interp->codec_error_registry,
86484
3055e61f1e66 Issue #1772673: The type of `char*` arguments now changed to `const char*`.
Serhiy Storchaka <storchaka@gmail.com>
parents: 85157
diff changeset
464 name, error);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
465 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
466
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
467 /* Lookup the error handling callback function registered under the
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
468 name error. As a special case NULL can be passed, in which case
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
469 the error handling callback for strict encoding will be returned. */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
470 PyObject *PyCodec_LookupError(const char *name)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
471 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
472 PyObject *handler = NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
473
31797
846b3ee7e106 Changed random calls to PyThreadState_Get() to use the macro
Nicholas Bastin <nick.bastin@gmail.com>
parents: 31273
diff changeset
474 PyInterpreterState *interp = PyThreadState_GET()->interp;
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
475 if (interp->codec_search_path == NULL && _PyCodecRegistry_Init())
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
476 return NULL;
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
477
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
478 if (name==NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
479 name = "strict";
86484
3055e61f1e66 Issue #1772673: The type of `char*` arguments now changed to `const char*`.
Serhiy Storchaka <storchaka@gmail.com>
parents: 85157
diff changeset
480 handler = PyDict_GetItemString(interp->codec_error_registry, name);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
481 if (!handler)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
482 PyErr_Format(PyExc_LookupError, "unknown error handler name '%.400s'", name);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
483 else
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
484 Py_INCREF(handler);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
485 return handler;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
486 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
487
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
488 static void wrong_exception_type(PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
489 {
72925
7109f31300fb Rename _Py_identifier to _Py_IDENTIFIER.
Martin v. Löwis <martin@v.loewis.de>
parents: 72918
diff changeset
490 _Py_IDENTIFIER(__class__);
7109f31300fb Rename _Py_identifier to _Py_IDENTIFIER.
Martin v. Löwis <martin@v.loewis.de>
parents: 72918
diff changeset
491 _Py_IDENTIFIER(__name__);
72848
81380082d216 Use identifier API for PyObject_GetAttrString.
Martin v. Löwis <martin@v.loewis.de>
parents: 72591
diff changeset
492 PyObject *type = _PyObject_GetAttrId(exc, &PyId___class__);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
493 if (type != NULL) {
72848
81380082d216 Use identifier API for PyObject_GetAttrString.
Martin v. Löwis <martin@v.loewis.de>
parents: 72591
diff changeset
494 PyObject *name = _PyObject_GetAttrId(type, &PyId___name__);
41878
089f449a20cd Change PyErr_Format() to generate a unicode string (by using
Walter Dörwald <walter@livinglogic.de>
parents: 41587
diff changeset
495 Py_DECREF(type);
089f449a20cd Change PyErr_Format() to generate a unicode string (by using
Walter Dörwald <walter@livinglogic.de>
parents: 41587
diff changeset
496 if (name != NULL) {
089f449a20cd Change PyErr_Format() to generate a unicode string (by using
Walter Dörwald <walter@livinglogic.de>
parents: 41587
diff changeset
497 PyErr_Format(PyExc_TypeError,
089f449a20cd Change PyErr_Format() to generate a unicode string (by using
Walter Dörwald <walter@livinglogic.de>
parents: 41587
diff changeset
498 "don't know how to handle %S in error callback", name);
089f449a20cd Change PyErr_Format() to generate a unicode string (by using
Walter Dörwald <walter@livinglogic.de>
parents: 41587
diff changeset
499 Py_DECREF(name);
089f449a20cd Change PyErr_Format() to generate a unicode string (by using
Walter Dörwald <walter@livinglogic.de>
parents: 41587
diff changeset
500 }
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
501 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
502 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
503
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
504 PyObject *PyCodec_StrictErrors(PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
505 {
36751
8e04de2480e4 PEP 352 implementation. Creates a new base class, BaseException, which has an
Brett Cannon <bcannon@gmail.com>
parents: 36712
diff changeset
506 if (PyExceptionInstance_Check(exc))
8e04de2480e4 PEP 352 implementation. Creates a new base class, BaseException, which has an
Brett Cannon <bcannon@gmail.com>
parents: 36712
diff changeset
507 PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
508 else
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
509 PyErr_SetString(PyExc_TypeError, "codec must pass exception instance");
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
510 return NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
511 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
512
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
513
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
514 PyObject *PyCodec_IgnoreErrors(PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
515 {
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 35939
diff changeset
516 Py_ssize_t end;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
517 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
518 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
519 return NULL;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
520 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
521 else if (PyObject_IsInstance(exc, PyExc_UnicodeDecodeError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
522 if (PyUnicodeDecodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
523 return NULL;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
524 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
525 else if (PyObject_IsInstance(exc, PyExc_UnicodeTranslateError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
526 if (PyUnicodeTranslateError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
527 return NULL;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
528 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
529 else {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
530 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
531 return NULL;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
532 }
73800
6407294eb3c6 PyCodec_IgnoreErrors() avoids the deprecated "u#" format
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73359
diff changeset
533 return Py_BuildValue("(Nn)", PyUnicode_New(0, 0), end);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
534 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
535
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
536
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
537 PyObject *PyCodec_ReplaceErrors(PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
538 {
72475
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
539 Py_ssize_t start, end, i, len;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
540
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
541 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
542 PyObject *res;
72475
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
543 int kind;
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
544 void *data;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
545 if (PyUnicodeEncodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
546 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
547 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
548 return NULL;
72475
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
549 len = end - start;
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
550 res = PyUnicode_New(len, '?');
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
551 if (res == NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
552 return NULL;
72475
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
553 kind = PyUnicode_KIND(res);
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
554 data = PyUnicode_DATA(res);
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
555 for (i = 0; i < len; ++i)
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
556 PyUnicode_WRITE(kind, data, i, '?');
76573
1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1)
Victor Stinner <victor.stinner@gmail.com>
parents: 74512
diff changeset
557 assert(_PyUnicode_CheckConsistency(res, 1));
72475
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
558 return Py_BuildValue("(Nn)", res, end);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
559 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
560 else if (PyObject_IsInstance(exc, PyExc_UnicodeDecodeError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
561 if (PyUnicodeDecodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
562 return NULL;
72591
611c57aa694a PyCodec_ReplaceErrors() uses "C" format instead of "u#" to build result
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72510
diff changeset
563 return Py_BuildValue("(Cn)",
611c57aa694a PyCodec_ReplaceErrors() uses "C" format instead of "u#" to build result
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72510
diff changeset
564 (int)Py_UNICODE_REPLACEMENT_CHARACTER,
611c57aa694a PyCodec_ReplaceErrors() uses "C" format instead of "u#" to build result
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72510
diff changeset
565 end);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
566 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
567 else if (PyObject_IsInstance(exc, PyExc_UnicodeTranslateError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
568 PyObject *res;
72475
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
569 int kind;
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
570 void *data;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
571 if (PyUnicodeTranslateError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
572 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
573 if (PyUnicodeTranslateError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
574 return NULL;
72475
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
575 len = end - start;
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
576 res = PyUnicode_New(len, Py_UNICODE_REPLACEMENT_CHARACTER);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
577 if (res == NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
578 return NULL;
72475
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
579 kind = PyUnicode_KIND(res);
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
580 data = PyUnicode_DATA(res);
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
581 for (i=0; i < len; i++)
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
582 PyUnicode_WRITE(kind, data, i, Py_UNICODE_REPLACEMENT_CHARACTER);
76573
1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1)
Victor Stinner <victor.stinner@gmail.com>
parents: 74512
diff changeset
583 assert(_PyUnicode_CheckConsistency(res, 1));
72475
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
584 return Py_BuildValue("(Nn)", res, end);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
585 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
586 else {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
587 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
588 return NULL;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
589 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
590 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
591
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
592 PyObject *PyCodec_XMLCharRefReplaceErrors(PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
593 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
594 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
595 PyObject *restuple;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
596 PyObject *object;
73358
2df7d749ee3a PyCodec_XMLCharRefReplaceError(): Remove unused variable
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73351
diff changeset
597 Py_ssize_t i;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
598 Py_ssize_t start;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
599 Py_ssize_t end;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
600 PyObject *res;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
601 unsigned char *outp;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
602 int ressize;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
603 Py_UCS4 ch;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
604 if (PyUnicodeEncodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
605 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
606 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
607 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
608 if (!(object = PyUnicodeEncodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
609 return NULL;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
610 for (i = start, ressize = 0; i < end; ++i) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
611 /* object is guaranteed to be "ready" */
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
612 ch = PyUnicode_READ_CHAR(object, i);
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
613 if (ch<10)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
614 ressize += 2+1+1;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
615 else if (ch<100)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
616 ressize += 2+2+1;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
617 else if (ch<1000)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
618 ressize += 2+3+1;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
619 else if (ch<10000)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
620 ressize += 2+4+1;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
621 else if (ch<100000)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
622 ressize += 2+5+1;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
623 else if (ch<1000000)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
624 ressize += 2+6+1;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
625 else
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
626 ressize += 2+7+1;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
627 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
628 /* allocate replacement */
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
629 res = PyUnicode_New(ressize, 127);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
630 if (res == NULL) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
631 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
632 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
633 }
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
634 outp = PyUnicode_1BYTE_DATA(res);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
635 /* generate replacement */
73358
2df7d749ee3a PyCodec_XMLCharRefReplaceError(): Remove unused variable
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73351
diff changeset
636 for (i = start; i < end; ++i) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
637 int digits;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
638 int base;
73351
2bec7c452b39 Fix C89 incompatibility.
Martin v. Löwis <martin@v.loewis.de>
parents: 73350
diff changeset
639 ch = PyUnicode_READ_CHAR(object, i);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
640 *outp++ = '&';
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
641 *outp++ = '#';
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
642 if (ch<10) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
643 digits = 1;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
644 base = 1;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
645 }
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
646 else if (ch<100) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
647 digits = 2;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
648 base = 10;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
649 }
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
650 else if (ch<1000) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
651 digits = 3;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
652 base = 100;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
653 }
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
654 else if (ch<10000) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
655 digits = 4;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
656 base = 1000;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
657 }
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
658 else if (ch<100000) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
659 digits = 5;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
660 base = 10000;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
661 }
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
662 else if (ch<1000000) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
663 digits = 6;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
664 base = 100000;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
665 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
666 else {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
667 digits = 7;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
668 base = 1000000;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
669 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
670 while (digits-->0) {
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
671 *outp++ = '0' + ch/base;
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
672 ch %= base;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
673 base /= 10;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
674 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
675 *outp++ = ';';
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
676 }
76573
1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1)
Victor Stinner <victor.stinner@gmail.com>
parents: 74512
diff changeset
677 assert(_PyUnicode_CheckConsistency(res, 1));
1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1)
Victor Stinner <victor.stinner@gmail.com>
parents: 74512
diff changeset
678 restuple = Py_BuildValue("(Nn)", res, end);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
679 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
680 return restuple;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
681 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
682 else {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
683 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
684 return NULL;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
685 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
686 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
687
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
688 PyObject *PyCodec_BackslashReplaceErrors(PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
689 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
690 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
691 PyObject *restuple;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
692 PyObject *object;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
693 Py_ssize_t i;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
694 Py_ssize_t start;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
695 Py_ssize_t end;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
696 PyObject *res;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
697 unsigned char *outp;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
698 int ressize;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
699 Py_UCS4 c;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
700 if (PyUnicodeEncodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
701 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
702 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
703 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
704 if (!(object = PyUnicodeEncodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
705 return NULL;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
706 for (i = start, ressize = 0; i < end; ++i) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
707 /* object is guaranteed to be "ready" */
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
708 c = PyUnicode_READ_CHAR(object, i);
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
709 if (c >= 0x10000) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
710 ressize += 1+1+8;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
711 }
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
712 else if (c >= 0x100) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
713 ressize += 1+1+4;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
714 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
715 else
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
716 ressize += 1+1+2;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
717 }
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
718 res = PyUnicode_New(ressize, 127);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
719 if (res==NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
720 return NULL;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
721 for (i = start, outp = PyUnicode_1BYTE_DATA(res);
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
722 i < end; ++i) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
723 c = PyUnicode_READ_CHAR(object, i);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
724 *outp++ = '\\';
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
725 if (c >= 0x00010000) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
726 *outp++ = 'U';
72918
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
727 *outp++ = Py_hexdigits[(c>>28)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
728 *outp++ = Py_hexdigits[(c>>24)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
729 *outp++ = Py_hexdigits[(c>>20)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
730 *outp++ = Py_hexdigits[(c>>16)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
731 *outp++ = Py_hexdigits[(c>>12)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
732 *outp++ = Py_hexdigits[(c>>8)&0xf];
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
733 }
64590
fc76e8c617dc Issue #9804: ascii() now always represents unicode surrogate pairs as
Antoine Pitrou <solipsis@pitrou.net>
parents: 61466
diff changeset
734 else if (c >= 0x100) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
735 *outp++ = 'u';
72918
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
736 *outp++ = Py_hexdigits[(c>>12)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
737 *outp++ = Py_hexdigits[(c>>8)&0xf];
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
738 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
739 else
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
740 *outp++ = 'x';
72918
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
741 *outp++ = Py_hexdigits[(c>>4)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
742 *outp++ = Py_hexdigits[c&0xf];
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
743 }
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
744
76573
1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1)
Victor Stinner <victor.stinner@gmail.com>
parents: 74512
diff changeset
745 assert(_PyUnicode_CheckConsistency(res, 1));
1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1)
Victor Stinner <victor.stinner@gmail.com>
parents: 74512
diff changeset
746 restuple = Py_BuildValue("(Nn)", res, end);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
747 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
748 return restuple;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
749 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
750 else {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
751 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
752 return NULL;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
753 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
754 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
755
87261
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
756 #define ENC_UTF8 0
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
757 #define ENC_UTF16BE 1
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
758 #define ENC_UTF16LE 2
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
759 #define ENC_UTF32BE 3
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
760 #define ENC_UTF32LE 4
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
761
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
762 static int
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
763 get_standard_encoding(const char *encoding, int *bytelength)
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
764 {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
765 if (Py_TOLOWER(encoding[0]) == 'u' &&
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
766 Py_TOLOWER(encoding[1]) == 't' &&
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
767 Py_TOLOWER(encoding[2]) == 'f') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
768 encoding += 3;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
769 if (*encoding == '-' || *encoding == '_' )
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
770 encoding++;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
771 if (encoding[0] == '1' && encoding[1] == '6') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
772 encoding += 2;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
773 *bytelength = 2;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
774 if (*encoding == '\0') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
775 #ifdef WORDS_BIGENDIAN
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
776 return ENC_UTF16BE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
777 #else
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
778 return ENC_UTF16LE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
779 #endif
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
780 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
781 if (*encoding == '-' || *encoding == '_' )
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
782 encoding++;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
783 if (Py_TOLOWER(encoding[1]) == 'e' && encoding[2] == '\0') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
784 if (Py_TOLOWER(encoding[0]) == 'b')
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
785 return ENC_UTF16BE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
786 if (Py_TOLOWER(encoding[0]) == 'l')
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
787 return ENC_UTF16LE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
788 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
789 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
790 else if (encoding[0] == '3' && encoding[1] == '2') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
791 encoding += 2;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
792 *bytelength = 4;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
793 if (*encoding == '\0') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
794 #ifdef WORDS_BIGENDIAN
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
795 return ENC_UTF32BE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
796 #else
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
797 return ENC_UTF32LE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
798 #endif
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
799 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
800 if (*encoding == '-' || *encoding == '_' )
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
801 encoding++;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
802 if (Py_TOLOWER(encoding[1]) == 'e' && encoding[2] == '\0') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
803 if (Py_TOLOWER(encoding[0]) == 'b')
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
804 return ENC_UTF32BE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
805 if (Py_TOLOWER(encoding[0]) == 'l')
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
806 return ENC_UTF32LE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
807 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
808 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
809 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
810 /* utf-8 */
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
811 *bytelength = 3;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
812 return ENC_UTF8;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
813 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
814
53687
52a89d723b6b Make PyCodec_SurrogateErrors static.
Martin v. Löwis <martin@v.loewis.de>
parents: 53681
diff changeset
815 /* This handler is declared static until someone demonstrates
52a89d723b6b Make PyCodec_SurrogateErrors static.
Martin v. Löwis <martin@v.loewis.de>
parents: 53681
diff changeset
816 a need to call it directly. */
52a89d723b6b Make PyCodec_SurrogateErrors static.
Martin v. Löwis <martin@v.loewis.de>
parents: 53681
diff changeset
817 static PyObject *
53965
fbcdae3e0ca1 Rename the surrogates error handler to surrogatepass.
Martin v. Löwis <martin@v.loewis.de>
parents: 53767
diff changeset
818 PyCodec_SurrogatePassErrors(PyObject *exc)
53681
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
819 {
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
820 PyObject *restuple;
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
821 PyObject *object;
87261
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
822 PyObject *encode;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
823 char *encoding;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
824 int code;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
825 int bytelength;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
826 Py_ssize_t i;
53681
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
827 Py_ssize_t start;
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
828 Py_ssize_t end;
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
829 PyObject *res;
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
830 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
87261
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
831 unsigned char *outp;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
832 if (PyUnicodeEncodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
833 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
834 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
835 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
836 if (!(object = PyUnicodeEncodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
837 return NULL;
87261
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
838 if (!(encode = PyUnicodeEncodeError_GetEncoding(exc))) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
839 Py_DECREF(object);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
840 return NULL;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
841 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
842 if (!(encoding = PyUnicode_AsUTF8(encode))) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
843 Py_DECREF(object);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
844 Py_DECREF(encode);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
845 return NULL;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
846 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
847 code = get_standard_encoding(encoding, &bytelength);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
848 Py_DECREF(encode);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
849
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
850 res = PyBytes_FromStringAndSize(NULL, bytelength*(end-start));
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
851 if (!res) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
852 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
853 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
854 }
87261
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
855 outp = (unsigned char*)PyBytes_AsString(res);
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
856 for (i = start; i < end; i++) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
857 /* object is guaranteed to be "ready" */
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
858 Py_UCS4 ch = PyUnicode_READ_CHAR(object, i);
80057
13dd8199c112 Issue #16330: Use surrogate-related macros
Victor Stinner <victor.stinner@gmail.com>
parents: 79954
diff changeset
859 if (!Py_UNICODE_IS_SURROGATE(ch)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
860 /* Not a surrogate, fail with original exception */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
861 PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
862 Py_DECREF(res);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
863 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
864 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
865 }
87261
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
866 switch (code) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
867 case ENC_UTF8:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
868 *outp++ = (unsigned char)(0xe0 | (ch >> 12));
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
869 *outp++ = (unsigned char)(0x80 | ((ch >> 6) & 0x3f));
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
870 *outp++ = (unsigned char)(0x80 | (ch & 0x3f));
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
871 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
872 case ENC_UTF16LE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
873 *outp++ = (unsigned char) ch;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
874 *outp++ = (unsigned char)(ch >> 8);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
875 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
876 case ENC_UTF16BE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
877 *outp++ = (unsigned char)(ch >> 8);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
878 *outp++ = (unsigned char) ch;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
879 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
880 case ENC_UTF32LE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
881 *outp++ = (unsigned char) ch;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
882 *outp++ = (unsigned char)(ch >> 8);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
883 *outp++ = (unsigned char)(ch >> 16);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
884 *outp++ = (unsigned char)(ch >> 24);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
885 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
886 case ENC_UTF32BE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
887 *outp++ = (unsigned char)(ch >> 24);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
888 *outp++ = (unsigned char)(ch >> 16);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
889 *outp++ = (unsigned char)(ch >> 8);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
890 *outp++ = (unsigned char) ch;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
891 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
892 }
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
893 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
894 restuple = Py_BuildValue("(On)", res, end);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
895 Py_DECREF(res);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
896 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
897 return restuple;
53681
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
898 }
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
899 else if (PyObject_IsInstance(exc, PyExc_UnicodeDecodeError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
900 unsigned char *p;
73359
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
901 Py_UCS4 ch = 0;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
902 if (PyUnicodeDecodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
903 return NULL;
87261
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
904 if (PyUnicodeDecodeError_GetEnd(exc, &end))
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
905 return NULL;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
906 if (!(object = PyUnicodeDecodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
907 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
908 if (!(p = (unsigned char*)PyBytes_AsString(object))) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
909 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
910 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
911 }
87261
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
912 if (!(encode = PyUnicodeDecodeError_GetEncoding(exc))) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
913 Py_DECREF(object);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
914 return NULL;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
915 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
916 if (!(encoding = PyUnicode_AsUTF8(encode))) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
917 Py_DECREF(object);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
918 Py_DECREF(encode);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
919 return NULL;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
920 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
921 code = get_standard_encoding(encoding, &bytelength);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
922 Py_DECREF(encode);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
923
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
924 /* Try decoding a single surrogate character. If
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
925 there are more, let the codec call us again. */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
926 p += start;
87261
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
927 if (PyBytes_GET_SIZE(object) - start >= bytelength) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
928 switch (code) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
929 case ENC_UTF8:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
930 if ((p[0] & 0xf0) == 0xe0 &&
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
931 (p[1] & 0xc0) == 0x80 &&
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
932 (p[2] & 0xc0) == 0x80) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
933 /* it's a three-byte code */
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
934 ch = ((p[0] & 0x0f) << 12) + ((p[1] & 0x3f) << 6) + (p[2] & 0x3f);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
935 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
936 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
937 case ENC_UTF16LE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
938 ch = p[1] << 8 | p[0];
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
939 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
940 case ENC_UTF16BE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
941 ch = p[0] << 8 | p[1];
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
942 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
943 case ENC_UTF32LE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
944 ch = (p[3] << 24) | (p[2] << 16) | (p[1] << 8) | p[0];
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
945 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
946 case ENC_UTF32BE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
947 ch = (p[0] << 24) | (p[1] << 16) | (p[2] << 8) | p[3];
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
948 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
949 }
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
950 }
87261
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
951
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
952 Py_DECREF(object);
87261
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
953 if (!Py_UNICODE_IS_SURROGATE(ch)) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
954 /* it's not a surrogate - fail */
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
955 PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
956 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
957 }
73359
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
958 res = PyUnicode_FromOrdinal(ch);
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
959 if (res == NULL)
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
960 return NULL;
87261
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
961 return Py_BuildValue("(Nn)", res, start + bytelength);
53681
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
962 }
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
963 else {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
964 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
965 return NULL;
53681
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
966 }
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
967 }
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
968
53767
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
969 static PyObject *
53966
7c29bfbfd352 Rename utf8b error handler to surrogateescape.
Martin v. Löwis <martin@v.loewis.de>
parents: 53965
diff changeset
970 PyCodec_SurrogateEscapeErrors(PyObject *exc)
53767
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
971 {
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
972 PyObject *restuple;
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
973 PyObject *object;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
974 Py_ssize_t i;
53767
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
975 Py_ssize_t start;
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
976 Py_ssize_t end;
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
977 PyObject *res;
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
978 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
979 char *outp;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
980 if (PyUnicodeEncodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
981 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
982 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
983 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
984 if (!(object = PyUnicodeEncodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
985 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
986 res = PyBytes_FromStringAndSize(NULL, end-start);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
987 if (!res) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
988 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
989 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
990 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
991 outp = PyBytes_AsString(res);
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
992 for (i = start; i < end; i++) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
993 /* object is guaranteed to be "ready" */
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
994 Py_UCS4 ch = PyUnicode_READ_CHAR(object, i);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
995 if (ch < 0xdc80 || ch > 0xdcff) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
996 /* Not a UTF-8b surrogate, fail with original exception */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
997 PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
998 Py_DECREF(res);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
999 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1000 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1001 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1002 *outp++ = ch - 0xdc00;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1003 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1004 restuple = Py_BuildValue("(On)", res, end);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1005 Py_DECREF(res);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1006 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1007 return restuple;
53767
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1008 }
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1009 else if (PyObject_IsInstance(exc, PyExc_UnicodeDecodeError)) {
73359
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
1010 PyObject *str;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1011 unsigned char *p;
73359
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
1012 Py_UCS2 ch[4]; /* decode up to 4 bad bytes. */
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1013 int consumed = 0;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1014 if (PyUnicodeDecodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1015 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1016 if (PyUnicodeDecodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1017 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1018 if (!(object = PyUnicodeDecodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1019 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1020 if (!(p = (unsigned char*)PyBytes_AsString(object))) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1021 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1022 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1023 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1024 while (consumed < 4 && consumed < end-start) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1025 /* Refuse to escape ASCII bytes. */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1026 if (p[start+consumed] < 128)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1027 break;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1028 ch[consumed] = 0xdc00 + p[start+consumed];
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1029 consumed++;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1030 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1031 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1032 if (!consumed) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1033 /* codec complained about ASCII byte. */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1034 PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1035 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1036 }
73359
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
1037 str = PyUnicode_FromKindAndData(PyUnicode_2BYTE_KIND, ch, consumed);
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
1038 if (str == NULL)
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
1039 return NULL;
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
1040 return Py_BuildValue("(Nn)", str, start+consumed);
53767
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1041 }
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1042 else {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1043 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1044 return NULL;
53767
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1045 }
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1046 }
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1047
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1048
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1049 static PyObject *strict_errors(PyObject *self, PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1050 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1051 return PyCodec_StrictErrors(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1052 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1053
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1054
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1055 static PyObject *ignore_errors(PyObject *self, PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1056 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1057 return PyCodec_IgnoreErrors(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1058 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1059
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1060
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1061 static PyObject *replace_errors(PyObject *self, PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1062 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1063 return PyCodec_ReplaceErrors(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1064 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1065
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1066
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1067 static PyObject *xmlcharrefreplace_errors(PyObject *self, PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1068 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1069 return PyCodec_XMLCharRefReplaceErrors(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1070 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1071
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1072
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1073 static PyObject *backslashreplace_errors(PyObject *self, PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1074 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1075 return PyCodec_BackslashReplaceErrors(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1076 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1077
53965
fbcdae3e0ca1 Rename the surrogates error handler to surrogatepass.
Martin v. Löwis <martin@v.loewis.de>
parents: 53767
diff changeset
1078 static PyObject *surrogatepass_errors(PyObject *self, PyObject *exc)
53681
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
1079 {
53965
fbcdae3e0ca1 Rename the surrogates error handler to surrogatepass.
Martin v. Löwis <martin@v.loewis.de>
parents: 53767
diff changeset
1080 return PyCodec_SurrogatePassErrors(exc);
53681
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
1081 }
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
1082
53966
7c29bfbfd352 Rename utf8b error handler to surrogateescape.
Martin v. Löwis <martin@v.loewis.de>
parents: 53965
diff changeset
1083 static PyObject *surrogateescape_errors(PyObject *self, PyObject *exc)
53767
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1084 {
53966
7c29bfbfd352 Rename utf8b error handler to surrogateescape.
Martin v. Löwis <martin@v.loewis.de>
parents: 53965
diff changeset
1085 return PyCodec_SurrogateEscapeErrors(exc);
53767
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1086 }
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1087
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1088 static int _PyCodecRegistry_Init(void)
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
1089 {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1090 static struct {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1091 char *name;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1092 PyMethodDef def;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1093 } methods[] =
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1094 {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1095 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1096 "strict",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1097 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1098 "strict_errors",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1099 strict_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1100 METH_O,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1101 PyDoc_STR("Implements the 'strict' error handling, which "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1102 "raises a UnicodeError on coding errors.")
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1103 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1104 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1105 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1106 "ignore",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1107 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1108 "ignore_errors",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1109 ignore_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1110 METH_O,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1111 PyDoc_STR("Implements the 'ignore' error handling, which "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1112 "ignores malformed data and continues.")
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1113 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1114 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1115 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1116 "replace",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1117 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1118 "replace_errors",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1119 replace_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1120 METH_O,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1121 PyDoc_STR("Implements the 'replace' error handling, which "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1122 "replaces malformed data with a replacement marker.")
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1123 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1124 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1125 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1126 "xmlcharrefreplace",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1127 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1128 "xmlcharrefreplace_errors",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1129 xmlcharrefreplace_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1130 METH_O,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1131 PyDoc_STR("Implements the 'xmlcharrefreplace' error handling, "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1132 "which replaces an unencodable character with the "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1133 "appropriate XML character reference.")
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1134 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1135 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1136 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1137 "backslashreplace",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1138 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1139 "backslashreplace_errors",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1140 backslashreplace_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1141 METH_O,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1142 PyDoc_STR("Implements the 'backslashreplace' error handling, "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1143 "which replaces an unencodable character with a "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1144 "backslashed escape sequence.")
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1145 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1146 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1147 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1148 "surrogatepass",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1149 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1150 "surrogatepass",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1151 surrogatepass_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1152 METH_O
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1153 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1154 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1155 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1156 "surrogateescape",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1157 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1158 "surrogateescape",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1159 surrogateescape_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1160 METH_O
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1161 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1162 }
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1163 };
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1164
31797
846b3ee7e106 Changed random calls to PyThreadState_Get() to use the macro
Nicholas Bastin <nick.bastin@gmail.com>
parents: 31273
diff changeset
1165 PyInterpreterState *interp = PyThreadState_GET()->interp;
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1166 PyObject *mod;
32533
201986d6d9d6 Fix a couple of signed/unsigned comparison warnings
Neal Norwitz <nnorwitz@gmail.com>
parents: 31797
diff changeset
1167 unsigned i;
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1168
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1169 if (interp->codec_search_path != NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1170 return 0;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1171
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1172 interp->codec_search_path = PyList_New(0);
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1173 interp->codec_search_cache = PyDict_New();
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1174 interp->codec_error_registry = PyDict_New();
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1175
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1176 if (interp->codec_error_registry) {
72510
b534b56379cb Use the new Py_ARRAY_LENGTH macro
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72475
diff changeset
1177 for (i = 0; i < Py_ARRAY_LENGTH(methods); ++i) {
81031
3a86a3f1d89a Issue #15422: get rid of PyCFunction_New macro
Andrew Svetlov <andrew.svetlov@gmail.com>
parents: 80218
diff changeset
1178 PyObject *func = PyCFunction_NewEx(&methods[i].def, NULL, NULL);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1179 int res;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1180 if (!func)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1181 Py_FatalError("can't initialize codec error registry");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1182 res = PyCodec_RegisterError(methods[i].name, func);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1183 Py_DECREF(func);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1184 if (res)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1185 Py_FatalError("can't initialize codec error registry");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1186 }
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1187 }
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1188
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1189 if (interp->codec_search_path == NULL ||
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1190 interp->codec_search_cache == NULL ||
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1191 interp->codec_error_registry == NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1192 Py_FatalError("can't initialize codec registry");
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
1193
44442
fa2f7183b1af More PyImport_ImportModule -> PyImport_ImportModuleNoBlock
Christian Heimes <christian@cheimes.de>
parents: 44351
diff changeset
1194 mod = PyImport_ImportModuleNoBlock("encodings");
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1195 if (mod == NULL) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1196 return -1;
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1197 }
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1198 Py_DECREF(mod);
49613
efef5fc45a9a Issue 3723: Fixed initialization of subinterpreters
Christian Heimes <christian@cheimes.de>
parents: 47496
diff changeset
1199 interp->codecs_initialized = 1;
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1200 return 0;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
1201 }