Mercurial > cpython-lru_cache-weakref

annotate Python/codecs.c @ 87643:2a679870d7d2

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
More realistic example for selectors.py.
author Guido van Rossum <guido@python.org>
date Fri, 29 Nov 2013 14:51:18 -0800
parents d68df99d7a57
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
1 /* ------------------------------------------------------------------------
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
2
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
3 Python Codec Registry and support functions
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
4
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
5 Written by Marc-Andre Lemburg (mal@lemburg.com).
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
6
13828
254f8b7c4a83 Changing the CNRI copyright notice according to CNRI's instructions.
Guido van Rossum <guido@python.org>
parents: 13658
diff changeset
7 Copyright (c) Corporation for National Research Initiatives.
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
8
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
9 ------------------------------------------------------------------------ */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
10
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
11 #include "Python.h"
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
12 #include <ctype.h>
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
13
72918
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
14 const char *Py_hexdigits = "0123456789abcdef";
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
15
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
16 /* --- Codec Registry ----------------------------------------------------- */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
17
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
18 /* Import the standard encodings package which will register the first
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
19 codec search function.
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
20
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
21 This is done in a lazy way so that the Unicode implementation does
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
22 not downgrade startup time of scripts not needing it.
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
23
12195
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
24 ImportErrors are silently ignored by this function. Only one try is
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
25 made.
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
26
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
27 */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
28
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
29 static int _PyCodecRegistry_Init(void); /* Forward */
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
30
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
31 int PyCodec_Register(PyObject *search_function)
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
32 {
31797
846b3ee7e106 Changed random calls to PyThreadState_Get() to use the macro
Nicholas Bastin <nick.bastin@gmail.com>
parents: 31273
diff changeset
33 PyInterpreterState *interp = PyThreadState_GET()->interp;
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
34 if (interp->codec_search_path == NULL && _PyCodecRegistry_Init())
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
35 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
36 if (search_function == NULL) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
37 PyErr_BadArgument();
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
38 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
39 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
40 if (!PyCallable_Check(search_function)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
41 PyErr_SetString(PyExc_TypeError, "argument must be callable");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
42 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
43 }
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
44 return PyList_Append(interp->codec_search_path, search_function);
12195
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
45
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
46 onError:
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
47 return -1;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
48 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
49
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
50 /* Convert a string to a normalized Python string: all characters are
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
51 converted to lower case, spaces are replaced with underscores. */
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
52
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
53 static
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
54 PyObject *normalizestring(const char *string)
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
55 {
85157
e7f6cef7a4cc Issue #18722: Remove uses of the "register" keyword in C code.
Antoine Pitrou <solipsis@pitrou.net>
parents: 84571
diff changeset
56 size_t i;
13096
8c8de5b736dc Trent Mick's Win64 changes: size_t vs. int or long; also some overflow
Guido van Rossum <guido@python.org>
parents: 12648
diff changeset
57 size_t len = strlen(string);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
58 char *p;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
59 PyObject *v;
43603
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
60
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
61 if (len > PY_SSIZE_T_MAX) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
62 PyErr_SetString(PyExc_OverflowError, "string is too large");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
63 return NULL;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
64 }
43603
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
65
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
66 p = PyMem_Malloc(len + 1);
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
67 if (p == NULL)
84571
4975bcd67aa7 Issue #18408: normalizestring() now raises MemoryError on memory allocation failure
Victor Stinner <victor.stinner@gmail.com>
parents: 81031
diff changeset
68 return PyErr_NoMemory();
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
69 for (i = 0; i < len; i++) {
85157
e7f6cef7a4cc Issue #18722: Remove uses of the "register" keyword in C code.
Antoine Pitrou <solipsis@pitrou.net>
parents: 84571
diff changeset
70 char ch = string[i];
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
71 if (ch == ' ')
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
72 ch = '-';
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
73 else
71489
92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales.
Antoine Pitrou <solipsis@pitrou.net>
parents: 64590
diff changeset
74 ch = Py_TOLOWER(Py_CHARMASK(ch));
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
75 p[i] = ch;
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
76 }
43603
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
77 p[i] = '\0';
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
78 v = PyUnicode_FromString(p);
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
79 if (v == NULL)
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
80 return NULL;
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
81 PyMem_Free(p);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
82 return v;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
83 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
84
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
85 /* Lookup the given encoding and return a tuple providing the codec
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
86 facilities.
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
87
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
88 The encoding string is looked up converted to all lower-case
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
89 characters. This makes encodings looked up through this mechanism
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
90 effectively case-insensitive.
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
91
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
92 If no codec is found, a LookupError is set and NULL returned.
12195
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
93
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
94 As side effect, this tries to load the encodings package, if not
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
95 yet done. This is part of the lazy load strategy for the encodings
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
96 package.
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
97
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
98 */
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
99
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
100 PyObject *_PyCodec_Lookup(const char *encoding)
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
101 {
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
102 PyInterpreterState *interp;
12048
f87b56eebacc Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12025
diff changeset
103 PyObject *result, *args = NULL, *v;
38453
29ae5e0fc348 Much-needed merge (using svnmerge.py this time) of trunk changes into p3yk.
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
104 Py_ssize_t i, len;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
105
12648
2a53f5dc0590 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12248
diff changeset
106 if (encoding == NULL) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
107 PyErr_BadArgument();
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
108 goto onError;
12648
2a53f5dc0590 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12248
diff changeset
109 }
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
110
31797
846b3ee7e106 Changed random calls to PyThreadState_Get() to use the macro
Nicholas Bastin <nick.bastin@gmail.com>
parents: 31273
diff changeset
111 interp = PyThreadState_GET()->interp;
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
112 if (interp->codec_search_path == NULL && _PyCodecRegistry_Init())
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
113 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
114
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
115 /* Convert the encoding to a normalized Python string: all
13620
5b727ab37092 Spelling fixes supplied by Rob W. W. Hooft. All these are fixes in either
Thomas Wouters <thomas@python.org>
parents: 13330
diff changeset
116 characters are converted to lower case, spaces and hyphens are
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
117 replaced with underscores. */
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
118 v = normalizestring(encoding);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
119 if (v == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
120 goto onError;
43603
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
121 PyUnicode_InternInPlace(&v);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
122
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
123 /* First, try to lookup the name in the registry dictionary */
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
124 result = PyDict_GetItem(interp->codec_search_cache, v);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
125 if (result != NULL) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
126 Py_INCREF(result);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
127 Py_DECREF(v);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
128 return result;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
129 }
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
130
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
131 /* Next, scan the search functions in order of registration */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
132 args = PyTuple_New(1);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
133 if (args == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
134 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
135 PyTuple_SET_ITEM(args,0,v);
12048
f87b56eebacc Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12025
diff changeset
136
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
137 len = PyList_Size(interp->codec_search_path);
12048
f87b56eebacc Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12025
diff changeset
138 if (len < 0)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
139 goto onError;
12195
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
140 if (len == 0) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
141 PyErr_SetString(PyExc_LookupError,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
142 "no codec search functions registered: "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
143 "can't find encoding");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
144 goto onError;
12195
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
145 }
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
146
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
147 for (i = 0; i < len; i++) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
148 PyObject *func;
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
149
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
150 func = PyList_GetItem(interp->codec_search_path, i);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
151 if (func == NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
152 goto onError;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
153 result = PyEval_CallObject(func, args);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
154 if (result == NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
155 goto onError;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
156 if (result == Py_None) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
157 Py_DECREF(result);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
158 continue;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
159 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
160 if (!PyTuple_Check(result) || PyTuple_GET_SIZE(result) != 4) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
161 PyErr_SetString(PyExc_TypeError,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
162 "codec search functions must return 4-tuples");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
163 Py_DECREF(result);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
164 goto onError;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
165 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
166 break;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
167 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
168 if (i == len) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
169 /* XXX Perhaps we should cache misses too ? */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
170 PyErr_Format(PyExc_LookupError,
25441
d32588c88633 Add encoding name in LookupError. Fixes #615013. Will backport to 2.2.
Martin v. Löwis <martin@v.loewis.de>
parents: 25095
diff changeset
171 "unknown encoding: %s", encoding);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
172 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
173 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
174
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
175 /* Cache and return the result */
42534
5f9e79d1e158 Handle error
Neal Norwitz <nnorwitz@gmail.com>
parents: 42057
diff changeset
176 if (PyDict_SetItem(interp->codec_search_cache, v, result) < 0) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
177 Py_DECREF(result);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
178 goto onError;
42534
5f9e79d1e158 Handle error
Neal Norwitz <nnorwitz@gmail.com>
parents: 42057
diff changeset
179 }
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
180 Py_DECREF(args);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
181 return result;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
182
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
183 onError:
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
184 Py_XDECREF(args);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
185 return NULL;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
186 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
187
47496
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
188 /* Codec registry encoding check API. */
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
189
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
190 int PyCodec_KnownEncoding(const char *encoding)
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
191 {
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
192 PyObject *codecs;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
193
47496
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
194 codecs = _PyCodec_Lookup(encoding);
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
195 if (!codecs) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
196 PyErr_Clear();
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
197 return 0;
47496
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
198 }
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
199 else {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
200 Py_DECREF(codecs);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
201 return 1;
47496
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
202 }
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
203 }
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
204
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
205 static
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
206 PyObject *args_tuple(PyObject *object,
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
207 const char *errors)
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
208 {
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
209 PyObject *args;
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
210
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
211 args = PyTuple_New(1 + (errors != NULL));
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
212 if (args == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
213 return NULL;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
214 Py_INCREF(object);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
215 PyTuple_SET_ITEM(args,0,object);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
216 if (errors) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
217 PyObject *v;
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
218
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
219 v = PyUnicode_FromString(errors);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
220 if (v == NULL) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
221 Py_DECREF(args);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
222 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
223 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
224 PyTuple_SET_ITEM(args, 1, v);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
225 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
226 return args;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
227 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
228
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
229 /* Helper function to get a codec item */
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
230
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
231 static
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
232 PyObject *codec_getitem(const char *encoding, int index)
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
233 {
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
234 PyObject *codecs;
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
235 PyObject *v;
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
236
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
237 codecs = _PyCodec_Lookup(encoding);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
238 if (codecs == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
239 return NULL;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
240 v = PyTuple_GET_ITEM(codecs, index);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
241 Py_DECREF(codecs);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
242 Py_INCREF(v);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
243 return v;
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
244 }
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
245
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
246 /* Helper function to create an incremental codec. */
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
247
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
248 static
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
249 PyObject *codec_getincrementalcodec(const char *encoding,
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
250 const char *errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
251 const char *attrname)
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
252 {
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
253 PyObject *codecs, *ret, *inccodec;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
254
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
255 codecs = _PyCodec_Lookup(encoding);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
256 if (codecs == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
257 return NULL;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
258 inccodec = PyObject_GetAttrString(codecs, attrname);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
259 Py_DECREF(codecs);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
260 if (inccodec == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
261 return NULL;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
262 if (errors)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
263 ret = PyObject_CallFunction(inccodec, "s", errors);
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
264 else
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
265 ret = PyObject_CallFunction(inccodec, NULL);
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
266 Py_DECREF(inccodec);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
267 return ret;
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
268 }
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
269
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
270 /* Helper function to create a stream codec. */
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
271
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
272 static
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
273 PyObject *codec_getstreamcodec(const char *encoding,
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
274 PyObject *stream,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
275 const char *errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
276 const int index)
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
277 {
39442
654c380cf8b9 Merged revisions 46753-51188 via svnmerge from
Thomas Wouters <thomas@python.org>
parents: 38453
diff changeset
278 PyObject *codecs, *streamcodec, *codeccls;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
279
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
280 codecs = _PyCodec_Lookup(encoding);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
281 if (codecs == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
282 return NULL;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
283
39442
654c380cf8b9 Merged revisions 46753-51188 via svnmerge from
Thomas Wouters <thomas@python.org>
parents: 38453
diff changeset
284 codeccls = PyTuple_GET_ITEM(codecs, index);
654c380cf8b9 Merged revisions 46753-51188 via svnmerge from
Thomas Wouters <thomas@python.org>
parents: 38453
diff changeset
285 if (errors != NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
286 streamcodec = PyObject_CallFunction(codeccls, "Os", stream, errors);
39442
654c380cf8b9 Merged revisions 46753-51188 via svnmerge from
Thomas Wouters <thomas@python.org>
parents: 38453
diff changeset
287 else
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
288 streamcodec = PyObject_CallFunction(codeccls, "O", stream);
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
289 Py_DECREF(codecs);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
290 return streamcodec;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
291 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
292
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
293 /* Convenience APIs to query the Codec registry.
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
294
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
295 All APIs return a codec object with incremented refcount.
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
296
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
297 */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
298
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
299 PyObject *PyCodec_Encoder(const char *encoding)
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
300 {
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
301 return codec_getitem(encoding, 0);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
302 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
303
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
304 PyObject *PyCodec_Decoder(const char *encoding)
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
305 {
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
306 return codec_getitem(encoding, 1);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
307 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
308
36996
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 36751
diff changeset
309 PyObject *PyCodec_IncrementalEncoder(const char *encoding,
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
310 const char *errors)
36996
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 36751
diff changeset
311 {
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
312 return codec_getincrementalcodec(encoding, errors, "incrementalencoder");
36996
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 36751
diff changeset
313 }
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 36751
diff changeset
314
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 36751
diff changeset
315 PyObject *PyCodec_IncrementalDecoder(const char *encoding,
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
316 const char *errors)
36996
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 36751
diff changeset
317 {
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
318 return codec_getincrementalcodec(encoding, errors, "incrementaldecoder");
36996
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 36751
diff changeset
319 }
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 36751
diff changeset
320
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
321 PyObject *PyCodec_StreamReader(const char *encoding,
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
322 PyObject *stream,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
323 const char *errors)
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
324 {
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
325 return codec_getstreamcodec(encoding, stream, errors, 2);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
326 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
327
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
328 PyObject *PyCodec_StreamWriter(const char *encoding,
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
329 PyObject *stream,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
330 const char *errors)
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
331 {
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
332 return codec_getstreamcodec(encoding, stream, errors, 3);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
333 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
334
87084
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
335 /* Helper that tries to ensure the reported exception chain indicates the
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
336 * codec that was invoked to trigger the failure without changing the type
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
337 * of the exception raised.
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
338 */
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
339 static void
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
340 wrap_codec_error(const char *operation,
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
341 const char *encoding)
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
342 {
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
343 /* TrySetFromCause will replace the active exception with a suitably
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
344 * updated clone if it can, otherwise it will leave the original
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
345 * exception alone.
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
346 */
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
347 _PyErr_TrySetFromCause("%s with '%s' codec failed",
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
348 operation, encoding);
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
349 }
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
350
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
351 /* Encode an object (e.g. an Unicode object) using the given encoding
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
352 and return the resulting encoded object (usually a Python string).
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
353
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
354 errors is passed to the encoder factory as argument if non-NULL. */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
355
87338
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
356 static PyObject *
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
357 _PyCodec_EncodeInternal(PyObject *object,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
358 PyObject *encoder,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
359 const char *encoding,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
360 const char *errors)
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
361 {
35939
79374eb6db51 Fix a few more ref leaks. Backport candidate
Neal Norwitz <nnorwitz@gmail.com>
parents: 32533
diff changeset
362 PyObject *args = NULL, *result = NULL;
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
363 PyObject *v = NULL;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
364
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
365 args = args_tuple(object, errors);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
366 if (args == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
367 goto onError;
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
368
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
369 result = PyEval_CallObject(encoder, args);
87109
4ea622c085ca Close 19609: narrow scope of codec exc chaining
Nick Coghlan <ncoghlan@gmail.com>
parents: 87084
diff changeset
370 if (result == NULL) {
4ea622c085ca Close 19609: narrow scope of codec exc chaining
Nick Coghlan <ncoghlan@gmail.com>
parents: 87084
diff changeset
371 wrap_codec_error("encoding", encoding);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
372 goto onError;
87109
4ea622c085ca Close 19609: narrow scope of codec exc chaining
Nick Coghlan <ncoghlan@gmail.com>
parents: 87084
diff changeset
373 }
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
374
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
375 if (!PyTuple_Check(result) ||
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
376 PyTuple_GET_SIZE(result) != 2) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
377 PyErr_SetString(PyExc_TypeError,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
378 "encoder must return a tuple (object, integer)");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
379 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
380 }
47496
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
381 v = PyTuple_GET_ITEM(result,0);
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
382 Py_INCREF(v);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
383 /* We don't check or use the second (integer) entry. */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
384
47496
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
385 Py_DECREF(args);
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
386 Py_DECREF(encoder);
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
387 Py_DECREF(result);
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
388 return v;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
389
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
390 onError:
35939
79374eb6db51 Fix a few more ref leaks. Backport candidate
Neal Norwitz <nnorwitz@gmail.com>
parents: 32533
diff changeset
391 Py_XDECREF(result);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
392 Py_XDECREF(args);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
393 Py_XDECREF(encoder);
47496
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
394 return NULL;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
395 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
396
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
397 /* Decode an object (usually a Python string) using the given encoding
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
398 and return an equivalent object (e.g. an Unicode object).
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
399
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
400 errors is passed to the decoder factory as argument if non-NULL. */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
401
87338
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
402 static PyObject *
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
403 _PyCodec_DecodeInternal(PyObject *object,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
404 PyObject *decoder,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
405 const char *encoding,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
406 const char *errors)
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
407 {
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
408 PyObject *args = NULL, *result = NULL;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
409 PyObject *v;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
410
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
411 args = args_tuple(object, errors);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
412 if (args == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
413 goto onError;
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
414
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
415 result = PyEval_CallObject(decoder,args);
87109
4ea622c085ca Close 19609: narrow scope of codec exc chaining
Nick Coghlan <ncoghlan@gmail.com>
parents: 87084
diff changeset
416 if (result == NULL) {
4ea622c085ca Close 19609: narrow scope of codec exc chaining
Nick Coghlan <ncoghlan@gmail.com>
parents: 87084
diff changeset
417 wrap_codec_error("decoding", encoding);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
418 goto onError;
87109
4ea622c085ca Close 19609: narrow scope of codec exc chaining
Nick Coghlan <ncoghlan@gmail.com>
parents: 87084
diff changeset
419 }
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
420 if (!PyTuple_Check(result) ||
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
421 PyTuple_GET_SIZE(result) != 2) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
422 PyErr_SetString(PyExc_TypeError,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
423 "decoder must return a tuple (object,integer)");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
424 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
425 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
426 v = PyTuple_GET_ITEM(result,0);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
427 Py_INCREF(v);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
428 /* We don't check or use the second (integer) entry. */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
429
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
430 Py_DECREF(args);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
431 Py_DECREF(decoder);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
432 Py_DECREF(result);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
433 return v;
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
434
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
435 onError:
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
436 Py_XDECREF(args);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
437 Py_XDECREF(decoder);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
438 Py_XDECREF(result);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
439 return NULL;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
440 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
441
87338
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
442 /* Generic encoding/decoding API */
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
443 PyObject *PyCodec_Encode(PyObject *object,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
444 const char *encoding,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
445 const char *errors)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
446 {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
447 PyObject *encoder;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
448
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
449 encoder = PyCodec_Encoder(encoding);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
450 if (encoder == NULL)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
451 return NULL;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
452
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
453 return _PyCodec_EncodeInternal(object, encoder, encoding, errors);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
454 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
455
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
456 PyObject *PyCodec_Decode(PyObject *object,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
457 const char *encoding,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
458 const char *errors)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
459 {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
460 PyObject *decoder;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
461
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
462 decoder = PyCodec_Decoder(encoding);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
463 if (decoder == NULL)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
464 return NULL;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
465
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
466 return _PyCodec_DecodeInternal(object, decoder, encoding, errors);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
467 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
468
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
469 /* Text encoding/decoding API */
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
470 static
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
471 PyObject *codec_getitem_checked(const char *encoding,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
472 const char *operation_name,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
473 int index)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
474 {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
475 _Py_IDENTIFIER(_is_text_encoding);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
476 PyObject *codec;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
477 PyObject *attr;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
478 PyObject *v;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
479 int is_text_codec;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
480
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
481 codec = _PyCodec_Lookup(encoding);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
482 if (codec == NULL)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
483 return NULL;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
484
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
485 /* Backwards compatibility: assume any raw tuple describes a text
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
486 * encoding, and the same for anything lacking the private
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
487 * attribute.
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
488 */
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
489 if (!PyTuple_CheckExact(codec)) {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
490 attr = _PyObject_GetAttrId(codec, &PyId__is_text_encoding);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
491 if (attr == NULL) {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
492 if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
493 PyErr_Clear();
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
494 } else {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
495 Py_DECREF(codec);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
496 return NULL;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
497 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
498 } else {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
499 is_text_codec = PyObject_IsTrue(attr);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
500 Py_DECREF(attr);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
501 if (!is_text_codec) {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
502 Py_DECREF(codec);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
503 PyErr_Format(PyExc_LookupError,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
504 "'%.400s' is not a text encoding; "
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
505 "use codecs.%s() to handle arbitrary codecs",
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
506 encoding, operation_name);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
507 return NULL;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
508 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
509 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
510 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
511
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
512 v = PyTuple_GET_ITEM(codec, index);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
513 Py_DECREF(codec);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
514 Py_INCREF(v);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
515 return v;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
516 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
517
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
518 static PyObject * _PyCodec_TextEncoder(const char *encoding)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
519 {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
520 return codec_getitem_checked(encoding, "encode", 0);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
521 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
522
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
523 static PyObject * _PyCodec_TextDecoder(const char *encoding)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
524 {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
525 return codec_getitem_checked(encoding, "decode", 1);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
526 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
527
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
528 PyObject *_PyCodec_EncodeText(PyObject *object,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
529 const char *encoding,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
530 const char *errors)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
531 {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
532 PyObject *encoder;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
533
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
534 encoder = _PyCodec_TextEncoder(encoding);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
535 if (encoder == NULL)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
536 return NULL;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
537
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
538 return _PyCodec_EncodeInternal(object, encoder, encoding, errors);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
539 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
540
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
541 PyObject *_PyCodec_DecodeText(PyObject *object,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
542 const char *encoding,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
543 const char *errors)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
544 {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
545 PyObject *decoder;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
546
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
547 decoder = _PyCodec_TextDecoder(encoding);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
548 if (decoder == NULL)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
549 return NULL;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
550
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
551 return _PyCodec_DecodeInternal(object, decoder, encoding, errors);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
552 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API
Nick Coghlan <ncoghlan@gmail.com>
parents: 87261
diff changeset
553
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
554 /* Register the error handling callback function error under the name
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
555 name. This function will be called by the codec when it encounters
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
556 an unencodable characters/undecodable bytes and doesn't know the
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
557 callback name, when name is specified as the error parameter
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
558 in the call to the encode/decode function.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
559 Return 0 on success, -1 on error */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
560 int PyCodec_RegisterError(const char *name, PyObject *error)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
561 {
31797
846b3ee7e106 Changed random calls to PyThreadState_Get() to use the macro
Nicholas Bastin <nick.bastin@gmail.com>
parents: 31273
diff changeset
562 PyInterpreterState *interp = PyThreadState_GET()->interp;
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
563 if (interp->codec_search_path == NULL && _PyCodecRegistry_Init())
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
564 return -1;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
565 if (!PyCallable_Check(error)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
566 PyErr_SetString(PyExc_TypeError, "handler must be callable");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
567 return -1;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
568 }
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
569 return PyDict_SetItemString(interp->codec_error_registry,
86484
3055e61f1e66 Issue #1772673: The type of `char*` arguments now changed to `const char*`.
Serhiy Storchaka <storchaka@gmail.com>
parents: 85157
diff changeset
570 name, error);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
571 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
572
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
573 /* Lookup the error handling callback function registered under the
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
574 name error. As a special case NULL can be passed, in which case
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
575 the error handling callback for strict encoding will be returned. */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
576 PyObject *PyCodec_LookupError(const char *name)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
577 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
578 PyObject *handler = NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
579
31797
846b3ee7e106 Changed random calls to PyThreadState_Get() to use the macro
Nicholas Bastin <nick.bastin@gmail.com>
parents: 31273
diff changeset
580 PyInterpreterState *interp = PyThreadState_GET()->interp;
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
581 if (interp->codec_search_path == NULL && _PyCodecRegistry_Init())
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
582 return NULL;
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
583
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
584 if (name==NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
585 name = "strict";
86484
3055e61f1e66 Issue #1772673: The type of `char*` arguments now changed to `const char*`.
Serhiy Storchaka <storchaka@gmail.com>
parents: 85157
diff changeset
586 handler = PyDict_GetItemString(interp->codec_error_registry, name);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
587 if (!handler)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
588 PyErr_Format(PyExc_LookupError, "unknown error handler name '%.400s'", name);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
589 else
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
590 Py_INCREF(handler);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
591 return handler;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
592 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
593
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
594 static void wrong_exception_type(PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
595 {
72925
7109f31300fb Rename _Py_identifier to _Py_IDENTIFIER.
Martin v. Löwis <martin@v.loewis.de>
parents: 72918
diff changeset
596 _Py_IDENTIFIER(__class__);
7109f31300fb Rename _Py_identifier to _Py_IDENTIFIER.
Martin v. Löwis <martin@v.loewis.de>
parents: 72918
diff changeset
597 _Py_IDENTIFIER(__name__);
72848
81380082d216 Use identifier API for PyObject_GetAttrString.
Martin v. Löwis <martin@v.loewis.de>
parents: 72591
diff changeset
598 PyObject *type = _PyObject_GetAttrId(exc, &PyId___class__);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
599 if (type != NULL) {
72848
81380082d216 Use identifier API for PyObject_GetAttrString.
Martin v. Löwis <martin@v.loewis.de>
parents: 72591
diff changeset
600 PyObject *name = _PyObject_GetAttrId(type, &PyId___name__);
41878
089f449a20cd Change PyErr_Format() to generate a unicode string (by using
Walter Dörwald <walter@livinglogic.de>
parents: 41587
diff changeset
601 Py_DECREF(type);
089f449a20cd Change PyErr_Format() to generate a unicode string (by using
Walter Dörwald <walter@livinglogic.de>
parents: 41587
diff changeset
602 if (name != NULL) {
089f449a20cd Change PyErr_Format() to generate a unicode string (by using
Walter Dörwald <walter@livinglogic.de>
parents: 41587
diff changeset
603 PyErr_Format(PyExc_TypeError,
089f449a20cd Change PyErr_Format() to generate a unicode string (by using
Walter Dörwald <walter@livinglogic.de>
parents: 41587
diff changeset
604 "don't know how to handle %S in error callback", name);
089f449a20cd Change PyErr_Format() to generate a unicode string (by using
Walter Dörwald <walter@livinglogic.de>
parents: 41587
diff changeset
605 Py_DECREF(name);
089f449a20cd Change PyErr_Format() to generate a unicode string (by using
Walter Dörwald <walter@livinglogic.de>
parents: 41587
diff changeset
606 }
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
607 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
608 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
609
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
610 PyObject *PyCodec_StrictErrors(PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
611 {
36751
8e04de2480e4 PEP 352 implementation. Creates a new base class, BaseException, which has an
Brett Cannon <bcannon@gmail.com>
parents: 36712
diff changeset
612 if (PyExceptionInstance_Check(exc))
8e04de2480e4 PEP 352 implementation. Creates a new base class, BaseException, which has an
Brett Cannon <bcannon@gmail.com>
parents: 36712
diff changeset
613 PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
614 else
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
615 PyErr_SetString(PyExc_TypeError, "codec must pass exception instance");
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
616 return NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
617 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
618
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
619
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
620 PyObject *PyCodec_IgnoreErrors(PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
621 {
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 35939
diff changeset
622 Py_ssize_t end;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
623 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
624 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
625 return NULL;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
626 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
627 else if (PyObject_IsInstance(exc, PyExc_UnicodeDecodeError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
628 if (PyUnicodeDecodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
629 return NULL;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
630 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
631 else if (PyObject_IsInstance(exc, PyExc_UnicodeTranslateError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
632 if (PyUnicodeTranslateError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
633 return NULL;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
634 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
635 else {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
636 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
637 return NULL;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
638 }
73800
6407294eb3c6 PyCodec_IgnoreErrors() avoids the deprecated "u#" format
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73359
diff changeset
639 return Py_BuildValue("(Nn)", PyUnicode_New(0, 0), end);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
640 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
641
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
642
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
643 PyObject *PyCodec_ReplaceErrors(PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
644 {
72475
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
645 Py_ssize_t start, end, i, len;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
646
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
647 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
648 PyObject *res;
72475
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
649 int kind;
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
650 void *data;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
651 if (PyUnicodeEncodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
652 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
653 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
654 return NULL;
72475
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
655 len = end - start;
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
656 res = PyUnicode_New(len, '?');
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
657 if (res == NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
658 return NULL;
72475
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
659 kind = PyUnicode_KIND(res);
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
660 data = PyUnicode_DATA(res);
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
661 for (i = 0; i < len; ++i)
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
662 PyUnicode_WRITE(kind, data, i, '?');
76573
1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1)
Victor Stinner <victor.stinner@gmail.com>
parents: 74512
diff changeset
663 assert(_PyUnicode_CheckConsistency(res, 1));
72475
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
664 return Py_BuildValue("(Nn)", res, end);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
665 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
666 else if (PyObject_IsInstance(exc, PyExc_UnicodeDecodeError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
667 if (PyUnicodeDecodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
668 return NULL;
72591
611c57aa694a PyCodec_ReplaceErrors() uses "C" format instead of "u#" to build result
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72510
diff changeset
669 return Py_BuildValue("(Cn)",
611c57aa694a PyCodec_ReplaceErrors() uses "C" format instead of "u#" to build result
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72510
diff changeset
670 (int)Py_UNICODE_REPLACEMENT_CHARACTER,
611c57aa694a PyCodec_ReplaceErrors() uses "C" format instead of "u#" to build result
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72510
diff changeset
671 end);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
672 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
673 else if (PyObject_IsInstance(exc, PyExc_UnicodeTranslateError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
674 PyObject *res;
72475
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
675 int kind;
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
676 void *data;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
677 if (PyUnicodeTranslateError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
678 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
679 if (PyUnicodeTranslateError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
680 return NULL;
72475
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
681 len = end - start;
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
682 res = PyUnicode_New(len, Py_UNICODE_REPLACEMENT_CHARACTER);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
683 if (res == NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
684 return NULL;
72475
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
685 kind = PyUnicode_KIND(res);
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
686 data = PyUnicode_DATA(res);
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
687 for (i=0; i < len; i++)
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
688 PyUnicode_WRITE(kind, data, i, Py_UNICODE_REPLACEMENT_CHARACTER);
76573
1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1)
Victor Stinner <victor.stinner@gmail.com>
parents: 74512
diff changeset
689 assert(_PyUnicode_CheckConsistency(res, 1));
72475
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
690 return Py_BuildValue("(Nn)", res, end);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
691 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
692 else {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
693 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
694 return NULL;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
695 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
696 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
697
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
698 PyObject *PyCodec_XMLCharRefReplaceErrors(PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
699 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
700 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
701 PyObject *restuple;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
702 PyObject *object;
73358
2df7d749ee3a PyCodec_XMLCharRefReplaceError(): Remove unused variable
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73351
diff changeset
703 Py_ssize_t i;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
704 Py_ssize_t start;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
705 Py_ssize_t end;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
706 PyObject *res;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
707 unsigned char *outp;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
708 int ressize;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
709 Py_UCS4 ch;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
710 if (PyUnicodeEncodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
711 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
712 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
713 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
714 if (!(object = PyUnicodeEncodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
715 return NULL;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
716 for (i = start, ressize = 0; i < end; ++i) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
717 /* object is guaranteed to be "ready" */
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
718 ch = PyUnicode_READ_CHAR(object, i);
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
719 if (ch<10)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
720 ressize += 2+1+1;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
721 else if (ch<100)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
722 ressize += 2+2+1;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
723 else if (ch<1000)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
724 ressize += 2+3+1;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
725 else if (ch<10000)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
726 ressize += 2+4+1;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
727 else if (ch<100000)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
728 ressize += 2+5+1;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
729 else if (ch<1000000)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
730 ressize += 2+6+1;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
731 else
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
732 ressize += 2+7+1;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
733 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
734 /* allocate replacement */
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
735 res = PyUnicode_New(ressize, 127);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
736 if (res == NULL) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
737 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
738 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
739 }
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
740 outp = PyUnicode_1BYTE_DATA(res);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
741 /* generate replacement */
73358
2df7d749ee3a PyCodec_XMLCharRefReplaceError(): Remove unused variable
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73351
diff changeset
742 for (i = start; i < end; ++i) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
743 int digits;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
744 int base;
73351
2bec7c452b39 Fix C89 incompatibility.
Martin v. Löwis <martin@v.loewis.de>
parents: 73350
diff changeset
745 ch = PyUnicode_READ_CHAR(object, i);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
746 *outp++ = '&';
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
747 *outp++ = '#';
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
748 if (ch<10) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
749 digits = 1;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
750 base = 1;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
751 }
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
752 else if (ch<100) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
753 digits = 2;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
754 base = 10;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
755 }
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
756 else if (ch<1000) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
757 digits = 3;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
758 base = 100;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
759 }
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
760 else if (ch<10000) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
761 digits = 4;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
762 base = 1000;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
763 }
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
764 else if (ch<100000) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
765 digits = 5;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
766 base = 10000;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
767 }
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
768 else if (ch<1000000) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
769 digits = 6;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
770 base = 100000;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
771 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
772 else {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
773 digits = 7;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
774 base = 1000000;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
775 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
776 while (digits-->0) {
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
777 *outp++ = '0' + ch/base;
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
778 ch %= base;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
779 base /= 10;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
780 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
781 *outp++ = ';';
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
782 }
76573
1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1)
Victor Stinner <victor.stinner@gmail.com>
parents: 74512
diff changeset
783 assert(_PyUnicode_CheckConsistency(res, 1));
1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1)
Victor Stinner <victor.stinner@gmail.com>
parents: 74512
diff changeset
784 restuple = Py_BuildValue("(Nn)", res, end);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
785 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
786 return restuple;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
787 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
788 else {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
789 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
790 return NULL;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
791 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
792 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
793
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
794 PyObject *PyCodec_BackslashReplaceErrors(PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
795 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
796 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
797 PyObject *restuple;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
798 PyObject *object;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
799 Py_ssize_t i;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
800 Py_ssize_t start;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
801 Py_ssize_t end;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
802 PyObject *res;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
803 unsigned char *outp;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
804 int ressize;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
805 Py_UCS4 c;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
806 if (PyUnicodeEncodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
807 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
808 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
809 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
810 if (!(object = PyUnicodeEncodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
811 return NULL;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
812 for (i = start, ressize = 0; i < end; ++i) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
813 /* object is guaranteed to be "ready" */
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
814 c = PyUnicode_READ_CHAR(object, i);
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
815 if (c >= 0x10000) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
816 ressize += 1+1+8;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
817 }
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
818 else if (c >= 0x100) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
819 ressize += 1+1+4;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
820 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
821 else
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
822 ressize += 1+1+2;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
823 }
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
824 res = PyUnicode_New(ressize, 127);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
825 if (res==NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
826 return NULL;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
827 for (i = start, outp = PyUnicode_1BYTE_DATA(res);
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
828 i < end; ++i) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
829 c = PyUnicode_READ_CHAR(object, i);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
830 *outp++ = '\\';
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
831 if (c >= 0x00010000) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
832 *outp++ = 'U';
72918
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
833 *outp++ = Py_hexdigits[(c>>28)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
834 *outp++ = Py_hexdigits[(c>>24)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
835 *outp++ = Py_hexdigits[(c>>20)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
836 *outp++ = Py_hexdigits[(c>>16)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
837 *outp++ = Py_hexdigits[(c>>12)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
838 *outp++ = Py_hexdigits[(c>>8)&0xf];
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
839 }
64590
fc76e8c617dc Issue #9804: ascii() now always represents unicode surrogate pairs as
Antoine Pitrou <solipsis@pitrou.net>
parents: 61466
diff changeset
840 else if (c >= 0x100) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
841 *outp++ = 'u';
72918
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
842 *outp++ = Py_hexdigits[(c>>12)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
843 *outp++ = Py_hexdigits[(c>>8)&0xf];
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
844 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
845 else
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
846 *outp++ = 'x';
72918
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
847 *outp++ = Py_hexdigits[(c>>4)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
848 *outp++ = Py_hexdigits[c&0xf];
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
849 }
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
850
76573
1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1)
Victor Stinner <victor.stinner@gmail.com>
parents: 74512
diff changeset
851 assert(_PyUnicode_CheckConsistency(res, 1));
1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1)
Victor Stinner <victor.stinner@gmail.com>
parents: 74512
diff changeset
852 restuple = Py_BuildValue("(Nn)", res, end);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
853 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
854 return restuple;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
855 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
856 else {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
857 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
858 return NULL;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
859 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
860 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
861
87261
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
862 #define ENC_UTF8 0
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
863 #define ENC_UTF16BE 1
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
864 #define ENC_UTF16LE 2
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
865 #define ENC_UTF32BE 3
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
866 #define ENC_UTF32LE 4
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
867
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
868 static int
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
869 get_standard_encoding(const char *encoding, int *bytelength)
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
870 {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
871 if (Py_TOLOWER(encoding[0]) == 'u' &&
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
872 Py_TOLOWER(encoding[1]) == 't' &&
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
873 Py_TOLOWER(encoding[2]) == 'f') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
874 encoding += 3;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
875 if (*encoding == '-' || *encoding == '_' )
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
876 encoding++;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
877 if (encoding[0] == '1' && encoding[1] == '6') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
878 encoding += 2;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
879 *bytelength = 2;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
880 if (*encoding == '\0') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
881 #ifdef WORDS_BIGENDIAN
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
882 return ENC_UTF16BE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
883 #else
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
884 return ENC_UTF16LE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
885 #endif
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
886 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
887 if (*encoding == '-' || *encoding == '_' )
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
888 encoding++;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
889 if (Py_TOLOWER(encoding[1]) == 'e' && encoding[2] == '\0') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
890 if (Py_TOLOWER(encoding[0]) == 'b')
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
891 return ENC_UTF16BE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
892 if (Py_TOLOWER(encoding[0]) == 'l')
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
893 return ENC_UTF16LE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
894 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
895 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
896 else if (encoding[0] == '3' && encoding[1] == '2') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
897 encoding += 2;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
898 *bytelength = 4;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
899 if (*encoding == '\0') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
900 #ifdef WORDS_BIGENDIAN
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
901 return ENC_UTF32BE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
902 #else
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
903 return ENC_UTF32LE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
904 #endif
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
905 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
906 if (*encoding == '-' || *encoding == '_' )
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
907 encoding++;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
908 if (Py_TOLOWER(encoding[1]) == 'e' && encoding[2] == '\0') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
909 if (Py_TOLOWER(encoding[0]) == 'b')
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
910 return ENC_UTF32BE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
911 if (Py_TOLOWER(encoding[0]) == 'l')
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
912 return ENC_UTF32LE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
913 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
914 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
915 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
916 /* utf-8 */
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
917 *bytelength = 3;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
918 return ENC_UTF8;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
919 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
920
53687
52a89d723b6b Make PyCodec_SurrogateErrors static.
Martin v. Löwis <martin@v.loewis.de>
parents: 53681
diff changeset
921 /* This handler is declared static until someone demonstrates
52a89d723b6b Make PyCodec_SurrogateErrors static.
Martin v. Löwis <martin@v.loewis.de>
parents: 53681
diff changeset
922 a need to call it directly. */
52a89d723b6b Make PyCodec_SurrogateErrors static.
Martin v. Löwis <martin@v.loewis.de>
parents: 53681
diff changeset
923 static PyObject *
53965
fbcdae3e0ca1 Rename the surrogates error handler to surrogatepass.
Martin v. Löwis <martin@v.loewis.de>
parents: 53767
diff changeset
924 PyCodec_SurrogatePassErrors(PyObject *exc)
53681
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
925 {
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
926 PyObject *restuple;
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
927 PyObject *object;
87261
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
928 PyObject *encode;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
929 char *encoding;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
930 int code;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
931 int bytelength;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
932 Py_ssize_t i;
53681
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
933 Py_ssize_t start;
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
934 Py_ssize_t end;
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
935 PyObject *res;
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
936 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
87261
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
937 unsigned char *outp;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
938 if (PyUnicodeEncodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
939 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
940 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
941 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
942 if (!(object = PyUnicodeEncodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
943 return NULL;
87261
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
944 if (!(encode = PyUnicodeEncodeError_GetEncoding(exc))) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
945 Py_DECREF(object);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
946 return NULL;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
947 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
948 if (!(encoding = PyUnicode_AsUTF8(encode))) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
949 Py_DECREF(object);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
950 Py_DECREF(encode);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
951 return NULL;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
952 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
953 code = get_standard_encoding(encoding, &bytelength);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
954 Py_DECREF(encode);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
955
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
956 res = PyBytes_FromStringAndSize(NULL, bytelength*(end-start));
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
957 if (!res) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
958 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
959 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
960 }
87261
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
961 outp = (unsigned char*)PyBytes_AsString(res);
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
962 for (i = start; i < end; i++) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
963 /* object is guaranteed to be "ready" */
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
964 Py_UCS4 ch = PyUnicode_READ_CHAR(object, i);
80057
13dd8199c112 Issue #16330: Use surrogate-related macros
Victor Stinner <victor.stinner@gmail.com>
parents: 79954
diff changeset
965 if (!Py_UNICODE_IS_SURROGATE(ch)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
966 /* Not a surrogate, fail with original exception */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
967 PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
968 Py_DECREF(res);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
969 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
970 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
971 }
87261
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
972 switch (code) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
973 case ENC_UTF8:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
974 *outp++ = (unsigned char)(0xe0 | (ch >> 12));
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
975 *outp++ = (unsigned char)(0x80 | ((ch >> 6) & 0x3f));
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
976 *outp++ = (unsigned char)(0x80 | (ch & 0x3f));
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
977 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
978 case ENC_UTF16LE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
979 *outp++ = (unsigned char) ch;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
980 *outp++ = (unsigned char)(ch >> 8);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
981 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
982 case ENC_UTF16BE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
983 *outp++ = (unsigned char)(ch >> 8);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
984 *outp++ = (unsigned char) ch;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
985 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
986 case ENC_UTF32LE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
987 *outp++ = (unsigned char) ch;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
988 *outp++ = (unsigned char)(ch >> 8);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
989 *outp++ = (unsigned char)(ch >> 16);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
990 *outp++ = (unsigned char)(ch >> 24);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
991 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
992 case ENC_UTF32BE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
993 *outp++ = (unsigned char)(ch >> 24);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
994 *outp++ = (unsigned char)(ch >> 16);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
995 *outp++ = (unsigned char)(ch >> 8);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
996 *outp++ = (unsigned char) ch;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
997 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
998 }
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
999 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1000 restuple = Py_BuildValue("(On)", res, end);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1001 Py_DECREF(res);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1002 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1003 return restuple;
53681
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
1004 }
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
1005 else if (PyObject_IsInstance(exc, PyExc_UnicodeDecodeError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1006 unsigned char *p;
73359
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
1007 Py_UCS4 ch = 0;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1008 if (PyUnicodeDecodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1009 return NULL;
87261
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1010 if (PyUnicodeDecodeError_GetEnd(exc, &end))
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1011 return NULL;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1012 if (!(object = PyUnicodeDecodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1013 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1014 if (!(p = (unsigned char*)PyBytes_AsString(object))) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1015 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1016 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1017 }
87261
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1018 if (!(encode = PyUnicodeDecodeError_GetEncoding(exc))) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1019 Py_DECREF(object);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1020 return NULL;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1021 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1022 if (!(encoding = PyUnicode_AsUTF8(encode))) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1023 Py_DECREF(object);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1024 Py_DECREF(encode);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1025 return NULL;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1026 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1027 code = get_standard_encoding(encoding, &bytelength);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1028 Py_DECREF(encode);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1029
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1030 /* Try decoding a single surrogate character. If
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1031 there are more, let the codec call us again. */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1032 p += start;
87261
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1033 if (PyBytes_GET_SIZE(object) - start >= bytelength) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1034 switch (code) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1035 case ENC_UTF8:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1036 if ((p[0] & 0xf0) == 0xe0 &&
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1037 (p[1] & 0xc0) == 0x80 &&
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1038 (p[2] & 0xc0) == 0x80) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1039 /* it's a three-byte code */
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1040 ch = ((p[0] & 0x0f) << 12) + ((p[1] & 0x3f) << 6) + (p[2] & 0x3f);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1041 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1042 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1043 case ENC_UTF16LE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1044 ch = p[1] << 8 | p[0];
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1045 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1046 case ENC_UTF16BE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1047 ch = p[0] << 8 | p[1];
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1048 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1049 case ENC_UTF32LE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1050 ch = (p[3] << 24) | (p[2] << 16) | (p[1] << 8) | p[0];
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1051 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1052 case ENC_UTF32BE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1053 ch = (p[0] << 24) | (p[1] << 16) | (p[2] << 8) | p[3];
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1054 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1055 }
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1056 }
87261
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1057
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1058 Py_DECREF(object);
87261
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1059 if (!Py_UNICODE_IS_SURROGATE(ch)) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1060 /* it's not a surrogate - fail */
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1061 PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1062 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1063 }
73359
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
1064 res = PyUnicode_FromOrdinal(ch);
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
1065 if (res == NULL)
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
1066 return NULL;
87261
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates.
Serhiy Storchaka <storchaka@gmail.com>
parents: 87109
diff changeset
1067 return Py_BuildValue("(Nn)", res, start + bytelength);
53681
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
1068 }
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
1069 else {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1070 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1071 return NULL;
53681
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
1072 }
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
1073 }
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
1074
53767
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1075 static PyObject *
53966
7c29bfbfd352 Rename utf8b error handler to surrogateescape.
Martin v. Löwis <martin@v.loewis.de>
parents: 53965
diff changeset
1076 PyCodec_SurrogateEscapeErrors(PyObject *exc)
53767
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1077 {
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1078 PyObject *restuple;
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1079 PyObject *object;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
1080 Py_ssize_t i;
53767
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1081 Py_ssize_t start;
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1082 Py_ssize_t end;
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1083 PyObject *res;
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1084 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1085 char *outp;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1086 if (PyUnicodeEncodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1087 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1088 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1089 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1090 if (!(object = PyUnicodeEncodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1091 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1092 res = PyBytes_FromStringAndSize(NULL, end-start);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1093 if (!res) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1094 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1095 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1096 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1097 outp = PyBytes_AsString(res);
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
1098 for (i = start; i < end; i++) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
1099 /* object is guaranteed to be "ready" */
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
1100 Py_UCS4 ch = PyUnicode_READ_CHAR(object, i);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1101 if (ch < 0xdc80 || ch > 0xdcff) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1102 /* Not a UTF-8b surrogate, fail with original exception */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1103 PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1104 Py_DECREF(res);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1105 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1106 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1107 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1108 *outp++ = ch - 0xdc00;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1109 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1110 restuple = Py_BuildValue("(On)", res, end);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1111 Py_DECREF(res);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1112 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1113 return restuple;
53767
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1114 }
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1115 else if (PyObject_IsInstance(exc, PyExc_UnicodeDecodeError)) {
73359
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
1116 PyObject *str;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1117 unsigned char *p;
73359
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
1118 Py_UCS2 ch[4]; /* decode up to 4 bad bytes. */
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1119 int consumed = 0;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1120 if (PyUnicodeDecodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1121 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1122 if (PyUnicodeDecodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1123 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1124 if (!(object = PyUnicodeDecodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1125 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1126 if (!(p = (unsigned char*)PyBytes_AsString(object))) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1127 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1128 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1129 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1130 while (consumed < 4 && consumed < end-start) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1131 /* Refuse to escape ASCII bytes. */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1132 if (p[start+consumed] < 128)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1133 break;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1134 ch[consumed] = 0xdc00 + p[start+consumed];
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1135 consumed++;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1136 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1137 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1138 if (!consumed) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1139 /* codec complained about ASCII byte. */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1140 PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1141 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1142 }
73359
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
1143 str = PyUnicode_FromKindAndData(PyUnicode_2BYTE_KIND, ch, consumed);
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
1144 if (str == NULL)
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
1145 return NULL;
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
1146 return Py_BuildValue("(Nn)", str, start+consumed);
53767
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1147 }
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1148 else {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1149 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1150 return NULL;
53767
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1151 }
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1152 }
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1153
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1154
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1155 static PyObject *strict_errors(PyObject *self, PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1156 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1157 return PyCodec_StrictErrors(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1158 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1159
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1160
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1161 static PyObject *ignore_errors(PyObject *self, PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1162 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1163 return PyCodec_IgnoreErrors(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1164 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1165
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1166
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1167 static PyObject *replace_errors(PyObject *self, PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1168 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1169 return PyCodec_ReplaceErrors(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1170 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1171
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1172
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1173 static PyObject *xmlcharrefreplace_errors(PyObject *self, PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1174 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1175 return PyCodec_XMLCharRefReplaceErrors(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1176 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1177
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1178
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1179 static PyObject *backslashreplace_errors(PyObject *self, PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1180 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1181 return PyCodec_BackslashReplaceErrors(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1182 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1183
53965
fbcdae3e0ca1 Rename the surrogates error handler to surrogatepass.
Martin v. Löwis <martin@v.loewis.de>
parents: 53767
diff changeset
1184 static PyObject *surrogatepass_errors(PyObject *self, PyObject *exc)
53681
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
1185 {
53965
fbcdae3e0ca1 Rename the surrogates error handler to surrogatepass.
Martin v. Löwis <martin@v.loewis.de>
parents: 53767
diff changeset
1186 return PyCodec_SurrogatePassErrors(exc);
53681
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
1187 }
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
1188
53966
7c29bfbfd352 Rename utf8b error handler to surrogateescape.
Martin v. Löwis <martin@v.loewis.de>
parents: 53965
diff changeset
1189 static PyObject *surrogateescape_errors(PyObject *self, PyObject *exc)
53767
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1190 {
53966
7c29bfbfd352 Rename utf8b error handler to surrogateescape.
Martin v. Löwis <martin@v.loewis.de>
parents: 53965
diff changeset
1191 return PyCodec_SurrogateEscapeErrors(exc);
53767
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1192 }
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
1193
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1194 static int _PyCodecRegistry_Init(void)
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
1195 {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1196 static struct {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1197 char *name;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1198 PyMethodDef def;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1199 } methods[] =
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1200 {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1201 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1202 "strict",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1203 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1204 "strict_errors",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1205 strict_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1206 METH_O,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1207 PyDoc_STR("Implements the 'strict' error handling, which "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1208 "raises a UnicodeError on coding errors.")
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1209 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1210 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1211 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1212 "ignore",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1213 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1214 "ignore_errors",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1215 ignore_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1216 METH_O,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1217 PyDoc_STR("Implements the 'ignore' error handling, which "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1218 "ignores malformed data and continues.")
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1219 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1220 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1221 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1222 "replace",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1223 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1224 "replace_errors",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1225 replace_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1226 METH_O,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1227 PyDoc_STR("Implements the 'replace' error handling, which "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1228 "replaces malformed data with a replacement marker.")
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1229 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1230 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1231 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1232 "xmlcharrefreplace",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1233 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1234 "xmlcharrefreplace_errors",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1235 xmlcharrefreplace_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1236 METH_O,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1237 PyDoc_STR("Implements the 'xmlcharrefreplace' error handling, "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1238 "which replaces an unencodable character with the "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1239 "appropriate XML character reference.")
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1240 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1241 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1242 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1243 "backslashreplace",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1244 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1245 "backslashreplace_errors",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1246 backslashreplace_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1247 METH_O,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1248 PyDoc_STR("Implements the 'backslashreplace' error handling, "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1249 "which replaces an unencodable character with a "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1250 "backslashed escape sequence.")
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1251 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1252 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1253 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1254 "surrogatepass",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1255 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1256 "surrogatepass",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1257 surrogatepass_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1258 METH_O
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1259 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1260 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1261 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1262 "surrogateescape",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1263 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1264 "surrogateescape",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1265 surrogateescape_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1266 METH_O
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1267 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1268 }
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1269 };
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1270
31797
846b3ee7e106 Changed random calls to PyThreadState_Get() to use the macro
Nicholas Bastin <nick.bastin@gmail.com>
parents: 31273
diff changeset
1271 PyInterpreterState *interp = PyThreadState_GET()->interp;
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1272 PyObject *mod;
32533
201986d6d9d6 Fix a couple of signed/unsigned comparison warnings
Neal Norwitz <nnorwitz@gmail.com>
parents: 31797
diff changeset
1273 unsigned i;
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1274
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1275 if (interp->codec_search_path != NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1276 return 0;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1277
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1278 interp->codec_search_path = PyList_New(0);
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1279 interp->codec_search_cache = PyDict_New();
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1280 interp->codec_error_registry = PyDict_New();
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1281
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1282 if (interp->codec_error_registry) {
72510
b534b56379cb Use the new Py_ARRAY_LENGTH macro
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72475
diff changeset
1283 for (i = 0; i < Py_ARRAY_LENGTH(methods); ++i) {
81031
3a86a3f1d89a Issue #15422: get rid of PyCFunction_New macro
Andrew Svetlov <andrew.svetlov@gmail.com>
parents: 80218
diff changeset
1284 PyObject *func = PyCFunction_NewEx(&methods[i].def, NULL, NULL);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1285 int res;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1286 if (!func)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1287 Py_FatalError("can't initialize codec error registry");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1288 res = PyCodec_RegisterError(methods[i].name, func);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1289 Py_DECREF(func);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1290 if (res)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1291 Py_FatalError("can't initialize codec error registry");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1292 }
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1293 }
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1294
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1295 if (interp->codec_search_path == NULL ||
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1296 interp->codec_search_cache == NULL ||
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1297 interp->codec_error_registry == NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1298 Py_FatalError("can't initialize codec registry");
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
1299
44442
fa2f7183b1af More PyImport_ImportModule -> PyImport_ImportModuleNoBlock
Christian Heimes <christian@cheimes.de>
parents: 44351
diff changeset
1300 mod = PyImport_ImportModuleNoBlock("encodings");
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1301 if (mod == NULL) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1302 return -1;
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1303 }
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1304 Py_DECREF(mod);
49613
efef5fc45a9a Issue 3723: Fixed initialization of subinterpreters
Christian Heimes <christian@cheimes.de>
parents: 47496
diff changeset
1305 interp->codecs_initialized = 1;
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1306 return 0;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
1307 }