Mercurial > cpython-lru_cache-weakref

annotate Python/codecs.c @ 87107:f7d401eaee0e

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Issue #19437: Fix init_builtin(), handle _PyImport_FindExtensionObject() failure
author Victor Stinner <victor.stinner@gmail.com>
date Thu, 14 Nov 2013 22:38:52 +0100
parents 854a2cea31b9
children 4ea622c085ca
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
1 /* ------------------------------------------------------------------------
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
2
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
3 Python Codec Registry and support functions
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
4
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
5 Written by Marc-Andre Lemburg (mal@lemburg.com).
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
6
13828
254f8b7c4a83 Changing the CNRI copyright notice according to CNRI's instructions.
Guido van Rossum <guido@python.org>
parents: 13658
diff changeset
7 Copyright (c) Corporation for National Research Initiatives.
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
8
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
9 ------------------------------------------------------------------------ */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
10
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
11 #include "Python.h"
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
12 #include <ctype.h>
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
13
72918
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
14 const char *Py_hexdigits = "0123456789abcdef";
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
15
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
16 /* --- Codec Registry ----------------------------------------------------- */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
17
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
18 /* Import the standard encodings package which will register the first
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
19 codec search function.
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
20
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
21 This is done in a lazy way so that the Unicode implementation does
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
22 not downgrade startup time of scripts not needing it.
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
23
12195
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
24 ImportErrors are silently ignored by this function. Only one try is
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
25 made.
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
26
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
27 */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
28
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
29 static int _PyCodecRegistry_Init(void); /* Forward */
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
30
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
31 int PyCodec_Register(PyObject *search_function)
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
32 {
31797
846b3ee7e106 Changed random calls to PyThreadState_Get() to use the macro
Nicholas Bastin <nick.bastin@gmail.com>
parents: 31273
diff changeset
33 PyInterpreterState *interp = PyThreadState_GET()->interp;
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
34 if (interp->codec_search_path == NULL && _PyCodecRegistry_Init())
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
35 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
36 if (search_function == NULL) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
37 PyErr_BadArgument();
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
38 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
39 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
40 if (!PyCallable_Check(search_function)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
41 PyErr_SetString(PyExc_TypeError, "argument must be callable");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
42 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
43 }
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
44 return PyList_Append(interp->codec_search_path, search_function);
12195
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
45
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
46 onError:
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
47 return -1;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
48 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
49
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
50 /* Convert a string to a normalized Python string: all characters are
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
51 converted to lower case, spaces are replaced with underscores. */
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
52
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
53 static
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
54 PyObject *normalizestring(const char *string)
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
55 {
85157
e7f6cef7a4cc Issue #18722: Remove uses of the "register" keyword in C code.
Antoine Pitrou <solipsis@pitrou.net>
parents: 84571
diff changeset
56 size_t i;
13096
8c8de5b736dc Trent Mick's Win64 changes: size_t vs. int or long; also some overflow
Guido van Rossum <guido@python.org>
parents: 12648
diff changeset
57 size_t len = strlen(string);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
58 char *p;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
59 PyObject *v;
43603
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
60
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
61 if (len > PY_SSIZE_T_MAX) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
62 PyErr_SetString(PyExc_OverflowError, "string is too large");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
63 return NULL;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
64 }
43603
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
65
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
66 p = PyMem_Malloc(len + 1);
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
67 if (p == NULL)
84571
4975bcd67aa7 Issue #18408: normalizestring() now raises MemoryError on memory allocation failure
Victor Stinner <victor.stinner@gmail.com>
parents: 81031
diff changeset
68 return PyErr_NoMemory();
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
69 for (i = 0; i < len; i++) {
85157
e7f6cef7a4cc Issue #18722: Remove uses of the "register" keyword in C code.
Antoine Pitrou <solipsis@pitrou.net>
parents: 84571
diff changeset
70 char ch = string[i];
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
71 if (ch == ' ')
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
72 ch = '-';
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
73 else
71489
92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales.
Antoine Pitrou <solipsis@pitrou.net>
parents: 64590
diff changeset
74 ch = Py_TOLOWER(Py_CHARMASK(ch));
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
75 p[i] = ch;
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
76 }
43603
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
77 p[i] = '\0';
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
78 v = PyUnicode_FromString(p);
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
79 if (v == NULL)
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
80 return NULL;
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
81 PyMem_Free(p);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
82 return v;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
83 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
84
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
85 /* Lookup the given encoding and return a tuple providing the codec
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
86 facilities.
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
87
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
88 The encoding string is looked up converted to all lower-case
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
89 characters. This makes encodings looked up through this mechanism
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
90 effectively case-insensitive.
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
91
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
92 If no codec is found, a LookupError is set and NULL returned.
12195
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
93
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
94 As side effect, this tries to load the encodings package, if not
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
95 yet done. This is part of the lazy load strategy for the encodings
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
96 package.
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
97
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
98 */
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
99
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
100 PyObject *_PyCodec_Lookup(const char *encoding)
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
101 {
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
102 PyInterpreterState *interp;
12048
f87b56eebacc Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12025
diff changeset
103 PyObject *result, *args = NULL, *v;
38453
29ae5e0fc348 Much-needed merge (using svnmerge.py this time) of trunk changes into p3yk.
Thomas Wouters <thomas@python.org>
parents: 37879
diff changeset
104 Py_ssize_t i, len;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
105
12648
2a53f5dc0590 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12248
diff changeset
106 if (encoding == NULL) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
107 PyErr_BadArgument();
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
108 goto onError;
12648
2a53f5dc0590 M.-A. Lemburg <mal@lemburg.com>:
Fred Drake <fdrake@acm.org>
parents: 12248
diff changeset
109 }
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
110
31797
846b3ee7e106 Changed random calls to PyThreadState_Get() to use the macro
Nicholas Bastin <nick.bastin@gmail.com>
parents: 31273
diff changeset
111 interp = PyThreadState_GET()->interp;
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
112 if (interp->codec_search_path == NULL && _PyCodecRegistry_Init())
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
113 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
114
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
115 /* Convert the encoding to a normalized Python string: all
13620
5b727ab37092 Spelling fixes supplied by Rob W. W. Hooft. All these are fixes in either
Thomas Wouters <thomas@python.org>
parents: 13330
diff changeset
116 characters are converted to lower case, spaces and hyphens are
12248
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
117 replaced with underscores. */
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that
Guido van Rossum <guido@python.org>
parents: 12195
diff changeset
118 v = normalizestring(encoding);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
119 if (v == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
120 goto onError;
43603
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee:
Guido van Rossum <guido@python.org>
parents: 42534
diff changeset
121 PyUnicode_InternInPlace(&v);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
122
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
123 /* First, try to lookup the name in the registry dictionary */
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
124 result = PyDict_GetItem(interp->codec_search_cache, v);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
125 if (result != NULL) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
126 Py_INCREF(result);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
127 Py_DECREF(v);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
128 return result;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
129 }
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
130
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
131 /* Next, scan the search functions in order of registration */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
132 args = PyTuple_New(1);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
133 if (args == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
134 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
135 PyTuple_SET_ITEM(args,0,v);
12048
f87b56eebacc Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12025
diff changeset
136
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
137 len = PyList_Size(interp->codec_search_path);
12048
f87b56eebacc Marc-Andre Lemburg:
Guido van Rossum <guido@python.org>
parents: 12025
diff changeset
138 if (len < 0)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
139 goto onError;
12195
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
140 if (len == 0) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
141 PyErr_SetString(PyExc_LookupError,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
142 "no codec search functions registered: "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
143 "can't find encoding");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
144 goto onError;
12195
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup
Guido van Rossum <guido@python.org>
parents: 12048
diff changeset
145 }
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
146
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
147 for (i = 0; i < len; i++) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
148 PyObject *func;
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
149
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
150 func = PyList_GetItem(interp->codec_search_path, i);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
151 if (func == NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
152 goto onError;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
153 result = PyEval_CallObject(func, args);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
154 if (result == NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
155 goto onError;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
156 if (result == Py_None) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
157 Py_DECREF(result);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
158 continue;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
159 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
160 if (!PyTuple_Check(result) || PyTuple_GET_SIZE(result) != 4) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
161 PyErr_SetString(PyExc_TypeError,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
162 "codec search functions must return 4-tuples");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
163 Py_DECREF(result);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
164 goto onError;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
165 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
166 break;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
167 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
168 if (i == len) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
169 /* XXX Perhaps we should cache misses too ? */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
170 PyErr_Format(PyExc_LookupError,
25441
d32588c88633 Add encoding name in LookupError. Fixes #615013. Will backport to 2.2.
Martin v. Löwis <martin@v.loewis.de>
parents: 25095
diff changeset
171 "unknown encoding: %s", encoding);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
172 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
173 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
174
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
175 /* Cache and return the result */
42534
5f9e79d1e158 Handle error
Neal Norwitz <nnorwitz@gmail.com>
parents: 42057
diff changeset
176 if (PyDict_SetItem(interp->codec_search_cache, v, result) < 0) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
177 Py_DECREF(result);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
178 goto onError;
42534
5f9e79d1e158 Handle error
Neal Norwitz <nnorwitz@gmail.com>
parents: 42057
diff changeset
179 }
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
180 Py_DECREF(args);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
181 return result;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
182
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
183 onError:
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
184 Py_XDECREF(args);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
185 return NULL;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
186 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
187
47496
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
188 /* Codec registry encoding check API. */
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
189
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
190 int PyCodec_KnownEncoding(const char *encoding)
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
191 {
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
192 PyObject *codecs;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
193
47496
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
194 codecs = _PyCodec_Lookup(encoding);
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
195 if (!codecs) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
196 PyErr_Clear();
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
197 return 0;
47496
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
198 }
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
199 else {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
200 Py_DECREF(codecs);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
201 return 1;
47496
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
202 }
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
203 }
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
204
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
205 static
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
206 PyObject *args_tuple(PyObject *object,
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
207 const char *errors)
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
208 {
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
209 PyObject *args;
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
210
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
211 args = PyTuple_New(1 + (errors != NULL));
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
212 if (args == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
213 return NULL;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
214 Py_INCREF(object);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
215 PyTuple_SET_ITEM(args,0,object);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
216 if (errors) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
217 PyObject *v;
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
218
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
219 v = PyUnicode_FromString(errors);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
220 if (v == NULL) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
221 Py_DECREF(args);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
222 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
223 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
224 PyTuple_SET_ITEM(args, 1, v);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
225 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
226 return args;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
227 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
228
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
229 /* Helper function to get a codec item */
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
230
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
231 static
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
232 PyObject *codec_getitem(const char *encoding, int index)
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
233 {
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
234 PyObject *codecs;
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
235 PyObject *v;
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
236
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
237 codecs = _PyCodec_Lookup(encoding);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
238 if (codecs == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
239 return NULL;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
240 v = PyTuple_GET_ITEM(codecs, index);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
241 Py_DECREF(codecs);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
242 Py_INCREF(v);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
243 return v;
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
244 }
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
245
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
246 /* Helper function to create an incremental codec. */
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
247
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
248 static
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
249 PyObject *codec_getincrementalcodec(const char *encoding,
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
250 const char *errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
251 const char *attrname)
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
252 {
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
253 PyObject *codecs, *ret, *inccodec;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
254
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
255 codecs = _PyCodec_Lookup(encoding);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
256 if (codecs == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
257 return NULL;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
258 inccodec = PyObject_GetAttrString(codecs, attrname);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
259 Py_DECREF(codecs);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
260 if (inccodec == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
261 return NULL;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
262 if (errors)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
263 ret = PyObject_CallFunction(inccodec, "s", errors);
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
264 else
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
265 ret = PyObject_CallFunction(inccodec, NULL);
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
266 Py_DECREF(inccodec);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
267 return ret;
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
268 }
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
269
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
270 /* Helper function to create a stream codec. */
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
271
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
272 static
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
273 PyObject *codec_getstreamcodec(const char *encoding,
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
274 PyObject *stream,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
275 const char *errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
276 const int index)
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
277 {
39442
654c380cf8b9 Merged revisions 46753-51188 via svnmerge from
Thomas Wouters <thomas@python.org>
parents: 38453
diff changeset
278 PyObject *codecs, *streamcodec, *codeccls;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
279
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
280 codecs = _PyCodec_Lookup(encoding);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
281 if (codecs == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
282 return NULL;
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
283
39442
654c380cf8b9 Merged revisions 46753-51188 via svnmerge from
Thomas Wouters <thomas@python.org>
parents: 38453
diff changeset
284 codeccls = PyTuple_GET_ITEM(codecs, index);
654c380cf8b9 Merged revisions 46753-51188 via svnmerge from
Thomas Wouters <thomas@python.org>
parents: 38453
diff changeset
285 if (errors != NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
286 streamcodec = PyObject_CallFunction(codeccls, "Os", stream, errors);
39442
654c380cf8b9 Merged revisions 46753-51188 via svnmerge from
Thomas Wouters <thomas@python.org>
parents: 38453
diff changeset
287 else
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
288 streamcodec = PyObject_CallFunction(codeccls, "O", stream);
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
289 Py_DECREF(codecs);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
290 return streamcodec;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
291 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
292
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
293 /* Convenience APIs to query the Codec registry.
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
294
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
295 All APIs return a codec object with incremented refcount.
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
296
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
297 */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
298
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
299 PyObject *PyCodec_Encoder(const char *encoding)
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
300 {
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
301 return codec_getitem(encoding, 0);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
302 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
303
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
304 PyObject *PyCodec_Decoder(const char *encoding)
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
305 {
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
306 return codec_getitem(encoding, 1);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
307 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
308
36996
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 36751
diff changeset
309 PyObject *PyCodec_IncrementalEncoder(const char *encoding,
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
310 const char *errors)
36996
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 36751
diff changeset
311 {
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
312 return codec_getincrementalcodec(encoding, errors, "incrementalencoder");
36996
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 36751
diff changeset
313 }
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 36751
diff changeset
314
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 36751
diff changeset
315 PyObject *PyCodec_IncrementalDecoder(const char *encoding,
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
316 const char *errors)
36996
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 36751
diff changeset
317 {
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
318 return codec_getincrementalcodec(encoding, errors, "incrementaldecoder");
36996
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 36751
diff changeset
319 }
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass
Walter Dörwald <walter@livinglogic.de>
parents: 36751
diff changeset
320
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
321 PyObject *PyCodec_StreamReader(const char *encoding,
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
322 PyObject *stream,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
323 const char *errors)
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
324 {
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
325 return codec_getstreamcodec(encoding, stream, errors, 2);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
326 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
327
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
328 PyObject *PyCodec_StreamWriter(const char *encoding,
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
329 PyObject *stream,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
330 const char *errors)
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
331 {
37879
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair
Thomas Wouters <thomas@python.org>
parents: 36996
diff changeset
332 return codec_getstreamcodec(encoding, stream, errors, 3);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
333 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
334
87084
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
335 /* Helper that tries to ensure the reported exception chain indicates the
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
336 * codec that was invoked to trigger the failure without changing the type
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
337 * of the exception raised.
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
338 */
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
339 static void
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
340 wrap_codec_error(const char *operation,
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
341 const char *encoding)
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
342 {
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
343 /* TrySetFromCause will replace the active exception with a suitably
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
344 * updated clone if it can, otherwise it will leave the original
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
345 * exception alone.
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
346 */
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
347 _PyErr_TrySetFromCause("%s with '%s' codec failed",
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
348 operation, encoding);
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
349 }
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
350
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
351 /* Encode an object (e.g. an Unicode object) using the given encoding
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
352 and return the resulting encoded object (usually a Python string).
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
353
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
354 errors is passed to the encoder factory as argument if non-NULL. */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
355
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
356 PyObject *PyCodec_Encode(PyObject *object,
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
357 const char *encoding,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
358 const char *errors)
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
359 {
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
360 PyObject *encoder = NULL;
35939
79374eb6db51 Fix a few more ref leaks. Backport candidate
Neal Norwitz <nnorwitz@gmail.com>
parents: 32533
diff changeset
361 PyObject *args = NULL, *result = NULL;
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
362 PyObject *v = NULL;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
363
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
364 encoder = PyCodec_Encoder(encoding);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
365 if (encoder == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
366 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
367
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
368 args = args_tuple(object, errors);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
369 if (args == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
370 goto onError;
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
371
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
372 result = PyEval_CallObject(encoder, args);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
373 if (result == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
374 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
375
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
376 if (!PyTuple_Check(result) ||
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
377 PyTuple_GET_SIZE(result) != 2) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
378 PyErr_SetString(PyExc_TypeError,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
379 "encoder must return a tuple (object, integer)");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
380 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
381 }
47496
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
382 v = PyTuple_GET_ITEM(result,0);
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
383 Py_INCREF(v);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
384 /* We don't check or use the second (integer) entry. */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
385
47496
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
386 Py_DECREF(args);
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
387 Py_DECREF(encoder);
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
388 Py_DECREF(result);
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
389 return v;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
390
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
391 onError:
35939
79374eb6db51 Fix a few more ref leaks. Backport candidate
Neal Norwitz <nnorwitz@gmail.com>
parents: 32533
diff changeset
392 Py_XDECREF(result);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
393 Py_XDECREF(args);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
394 Py_XDECREF(encoder);
87084
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
395 wrap_codec_error("encoding", encoding);
47496
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode().
Marc-André Lemburg <mal@egenix.com>
parents: 47301
diff changeset
396 return NULL;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
397 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
398
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
399 /* Decode an object (usually a Python string) using the given encoding
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
400 and return an equivalent object (e.g. an Unicode object).
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
401
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
402 errors is passed to the decoder factory as argument if non-NULL. */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
403
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
404 PyObject *PyCodec_Decode(PyObject *object,
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
405 const char *encoding,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
406 const char *errors)
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
407 {
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
408 PyObject *decoder = NULL;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
409 PyObject *args = NULL, *result = NULL;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
410 PyObject *v;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
411
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
412 decoder = PyCodec_Decoder(encoding);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
413 if (decoder == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
414 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
415
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
416 args = args_tuple(object, errors);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
417 if (args == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
418 goto onError;
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
419
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
420 result = PyEval_CallObject(decoder,args);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
421 if (result == NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
422 goto onError;
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
423 if (!PyTuple_Check(result) ||
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
424 PyTuple_GET_SIZE(result) != 2) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
425 PyErr_SetString(PyExc_TypeError,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
426 "decoder must return a tuple (object,integer)");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
427 goto onError;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
428 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
429 v = PyTuple_GET_ITEM(result,0);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
430 Py_INCREF(v);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
431 /* We don't check or use the second (integer) entry. */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
432
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
433 Py_DECREF(args);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
434 Py_DECREF(decoder);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
435 Py_DECREF(result);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
436 return v;
43777
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch.
Guido van Rossum <guido@python.org>
parents: 43603
diff changeset
437
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
438 onError:
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
439 Py_XDECREF(args);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
440 Py_XDECREF(decoder);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
441 Py_XDECREF(result);
87084
854a2cea31b9 Close #17828: better handling of codec errors
Nick Coghlan <ncoghlan@gmail.com>
parents: 86484
diff changeset
442 wrap_codec_error("decoding", encoding);
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
443 return NULL;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
444 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
445
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
446 /* Register the error handling callback function error under the name
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
447 name. This function will be called by the codec when it encounters
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
448 an unencodable characters/undecodable bytes and doesn't know the
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
449 callback name, when name is specified as the error parameter
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
450 in the call to the encode/decode function.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
451 Return 0 on success, -1 on error */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
452 int PyCodec_RegisterError(const char *name, PyObject *error)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
453 {
31797
846b3ee7e106 Changed random calls to PyThreadState_Get() to use the macro
Nicholas Bastin <nick.bastin@gmail.com>
parents: 31273
diff changeset
454 PyInterpreterState *interp = PyThreadState_GET()->interp;
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
455 if (interp->codec_search_path == NULL && _PyCodecRegistry_Init())
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
456 return -1;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
457 if (!PyCallable_Check(error)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
458 PyErr_SetString(PyExc_TypeError, "handler must be callable");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
459 return -1;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
460 }
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
461 return PyDict_SetItemString(interp->codec_error_registry,
86484
3055e61f1e66 Issue #1772673: The type of `char*` arguments now changed to `const char*`.
Serhiy Storchaka <storchaka@gmail.com>
parents: 85157
diff changeset
462 name, error);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
463 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
464
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
465 /* Lookup the error handling callback function registered under the
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
466 name error. As a special case NULL can be passed, in which case
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
467 the error handling callback for strict encoding will be returned. */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
468 PyObject *PyCodec_LookupError(const char *name)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
469 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
470 PyObject *handler = NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
471
31797
846b3ee7e106 Changed random calls to PyThreadState_Get() to use the macro
Nicholas Bastin <nick.bastin@gmail.com>
parents: 31273
diff changeset
472 PyInterpreterState *interp = PyThreadState_GET()->interp;
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
473 if (interp->codec_search_path == NULL && _PyCodecRegistry_Init())
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
474 return NULL;
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
475
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
476 if (name==NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
477 name = "strict";
86484
3055e61f1e66 Issue #1772673: The type of `char*` arguments now changed to `const char*`.
Serhiy Storchaka <storchaka@gmail.com>
parents: 85157
diff changeset
478 handler = PyDict_GetItemString(interp->codec_error_registry, name);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
479 if (!handler)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
480 PyErr_Format(PyExc_LookupError, "unknown error handler name '%.400s'", name);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
481 else
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
482 Py_INCREF(handler);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
483 return handler;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
484 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
485
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
486 static void wrong_exception_type(PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
487 {
72925
7109f31300fb Rename _Py_identifier to _Py_IDENTIFIER.
Martin v. Löwis <martin@v.loewis.de>
parents: 72918
diff changeset
488 _Py_IDENTIFIER(__class__);
7109f31300fb Rename _Py_identifier to _Py_IDENTIFIER.
Martin v. Löwis <martin@v.loewis.de>
parents: 72918
diff changeset
489 _Py_IDENTIFIER(__name__);
72848
81380082d216 Use identifier API for PyObject_GetAttrString.
Martin v. Löwis <martin@v.loewis.de>
parents: 72591
diff changeset
490 PyObject *type = _PyObject_GetAttrId(exc, &PyId___class__);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
491 if (type != NULL) {
72848
81380082d216 Use identifier API for PyObject_GetAttrString.
Martin v. Löwis <martin@v.loewis.de>
parents: 72591
diff changeset
492 PyObject *name = _PyObject_GetAttrId(type, &PyId___name__);
41878
089f449a20cd Change PyErr_Format() to generate a unicode string (by using
Walter Dörwald <walter@livinglogic.de>
parents: 41587
diff changeset
493 Py_DECREF(type);
089f449a20cd Change PyErr_Format() to generate a unicode string (by using
Walter Dörwald <walter@livinglogic.de>
parents: 41587
diff changeset
494 if (name != NULL) {
089f449a20cd Change PyErr_Format() to generate a unicode string (by using
Walter Dörwald <walter@livinglogic.de>
parents: 41587
diff changeset
495 PyErr_Format(PyExc_TypeError,
089f449a20cd Change PyErr_Format() to generate a unicode string (by using
Walter Dörwald <walter@livinglogic.de>
parents: 41587
diff changeset
496 "don't know how to handle %S in error callback", name);
089f449a20cd Change PyErr_Format() to generate a unicode string (by using
Walter Dörwald <walter@livinglogic.de>
parents: 41587
diff changeset
497 Py_DECREF(name);
089f449a20cd Change PyErr_Format() to generate a unicode string (by using
Walter Dörwald <walter@livinglogic.de>
parents: 41587
diff changeset
498 }
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
499 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
500 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
501
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
502 PyObject *PyCodec_StrictErrors(PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
503 {
36751
8e04de2480e4 PEP 352 implementation. Creates a new base class, BaseException, which has an
Brett Cannon <bcannon@gmail.com>
parents: 36712
diff changeset
504 if (PyExceptionInstance_Check(exc))
8e04de2480e4 PEP 352 implementation. Creates a new base class, BaseException, which has an
Brett Cannon <bcannon@gmail.com>
parents: 36712
diff changeset
505 PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
506 else
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
507 PyErr_SetString(PyExc_TypeError, "codec must pass exception instance");
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
508 return NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
509 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
510
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
511
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
512 PyObject *PyCodec_IgnoreErrors(PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
513 {
36501
7fdc639bc5b4 Merge ssize_t branch.
Martin v. Löwis <martin@v.loewis.de>
parents: 35939
diff changeset
514 Py_ssize_t end;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
515 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
516 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
517 return NULL;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
518 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
519 else if (PyObject_IsInstance(exc, PyExc_UnicodeDecodeError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
520 if (PyUnicodeDecodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
521 return NULL;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
522 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
523 else if (PyObject_IsInstance(exc, PyExc_UnicodeTranslateError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
524 if (PyUnicodeTranslateError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
525 return NULL;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
526 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
527 else {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
528 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
529 return NULL;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
530 }
73800
6407294eb3c6 PyCodec_IgnoreErrors() avoids the deprecated "u#" format
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73359
diff changeset
531 return Py_BuildValue("(Nn)", PyUnicode_New(0, 0), end);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
532 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
533
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
534
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
535 PyObject *PyCodec_ReplaceErrors(PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
536 {
72475
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
537 Py_ssize_t start, end, i, len;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
538
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
539 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
540 PyObject *res;
72475
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
541 int kind;
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
542 void *data;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
543 if (PyUnicodeEncodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
544 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
545 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
546 return NULL;
72475
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
547 len = end - start;
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
548 res = PyUnicode_New(len, '?');
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
549 if (res == NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
550 return NULL;
72475
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
551 kind = PyUnicode_KIND(res);
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
552 data = PyUnicode_DATA(res);
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
553 for (i = 0; i < len; ++i)
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
554 PyUnicode_WRITE(kind, data, i, '?');
76573
1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1)
Victor Stinner <victor.stinner@gmail.com>
parents: 74512
diff changeset
555 assert(_PyUnicode_CheckConsistency(res, 1));
72475
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
556 return Py_BuildValue("(Nn)", res, end);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
557 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
558 else if (PyObject_IsInstance(exc, PyExc_UnicodeDecodeError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
559 if (PyUnicodeDecodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
560 return NULL;
72591
611c57aa694a PyCodec_ReplaceErrors() uses "C" format instead of "u#" to build result
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72510
diff changeset
561 return Py_BuildValue("(Cn)",
611c57aa694a PyCodec_ReplaceErrors() uses "C" format instead of "u#" to build result
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72510
diff changeset
562 (int)Py_UNICODE_REPLACEMENT_CHARACTER,
611c57aa694a PyCodec_ReplaceErrors() uses "C" format instead of "u#" to build result
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72510
diff changeset
563 end);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
564 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
565 else if (PyObject_IsInstance(exc, PyExc_UnicodeTranslateError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
566 PyObject *res;
72475
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
567 int kind;
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
568 void *data;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
569 if (PyUnicodeTranslateError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
570 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
571 if (PyUnicodeTranslateError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
572 return NULL;
72475
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
573 len = end - start;
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
574 res = PyUnicode_New(len, Py_UNICODE_REPLACEMENT_CHARACTER);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
575 if (res == NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
576 return NULL;
72475
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
577 kind = PyUnicode_KIND(res);
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
578 data = PyUnicode_DATA(res);
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
579 for (i=0; i < len; i++)
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
580 PyUnicode_WRITE(kind, data, i, Py_UNICODE_REPLACEMENT_CHARACTER);
76573
1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1)
Victor Stinner <victor.stinner@gmail.com>
parents: 74512
diff changeset
581 assert(_PyUnicode_CheckConsistency(res, 1));
72475
8beaa9a37387 Implement PEP 393.
Martin v. Löwis <martin@v.loewis.de>
parents: 71489
diff changeset
582 return Py_BuildValue("(Nn)", res, end);
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
583 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
584 else {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
585 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
586 return NULL;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
587 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
588 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
589
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
590 PyObject *PyCodec_XMLCharRefReplaceErrors(PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
591 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
592 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
593 PyObject *restuple;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
594 PyObject *object;
73358
2df7d749ee3a PyCodec_XMLCharRefReplaceError(): Remove unused variable
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73351
diff changeset
595 Py_ssize_t i;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
596 Py_ssize_t start;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
597 Py_ssize_t end;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
598 PyObject *res;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
599 unsigned char *outp;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
600 int ressize;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
601 Py_UCS4 ch;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
602 if (PyUnicodeEncodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
603 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
604 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
605 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
606 if (!(object = PyUnicodeEncodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
607 return NULL;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
608 for (i = start, ressize = 0; i < end; ++i) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
609 /* object is guaranteed to be "ready" */
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
610 ch = PyUnicode_READ_CHAR(object, i);
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
611 if (ch<10)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
612 ressize += 2+1+1;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
613 else if (ch<100)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
614 ressize += 2+2+1;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
615 else if (ch<1000)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
616 ressize += 2+3+1;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
617 else if (ch<10000)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
618 ressize += 2+4+1;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
619 else if (ch<100000)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
620 ressize += 2+5+1;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
621 else if (ch<1000000)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
622 ressize += 2+6+1;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
623 else
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
624 ressize += 2+7+1;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
625 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
626 /* allocate replacement */
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
627 res = PyUnicode_New(ressize, 127);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
628 if (res == NULL) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
629 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
630 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
631 }
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
632 outp = PyUnicode_1BYTE_DATA(res);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
633 /* generate replacement */
73358
2df7d749ee3a PyCodec_XMLCharRefReplaceError(): Remove unused variable
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73351
diff changeset
634 for (i = start; i < end; ++i) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
635 int digits;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
636 int base;
73351
2bec7c452b39 Fix C89 incompatibility.
Martin v. Löwis <martin@v.loewis.de>
parents: 73350
diff changeset
637 ch = PyUnicode_READ_CHAR(object, i);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
638 *outp++ = '&';
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
639 *outp++ = '#';
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
640 if (ch<10) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
641 digits = 1;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
642 base = 1;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
643 }
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
644 else if (ch<100) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
645 digits = 2;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
646 base = 10;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
647 }
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
648 else if (ch<1000) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
649 digits = 3;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
650 base = 100;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
651 }
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
652 else if (ch<10000) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
653 digits = 4;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
654 base = 1000;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
655 }
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
656 else if (ch<100000) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
657 digits = 5;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
658 base = 10000;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
659 }
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
660 else if (ch<1000000) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
661 digits = 6;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
662 base = 100000;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
663 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
664 else {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
665 digits = 7;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
666 base = 1000000;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
667 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
668 while (digits-->0) {
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
669 *outp++ = '0' + ch/base;
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
670 ch %= base;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
671 base /= 10;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
672 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
673 *outp++ = ';';
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
674 }
76573
1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1)
Victor Stinner <victor.stinner@gmail.com>
parents: 74512
diff changeset
675 assert(_PyUnicode_CheckConsistency(res, 1));
1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1)
Victor Stinner <victor.stinner@gmail.com>
parents: 74512
diff changeset
676 restuple = Py_BuildValue("(Nn)", res, end);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
677 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
678 return restuple;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
679 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
680 else {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
681 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
682 return NULL;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
683 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
684 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
685
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
686 PyObject *PyCodec_BackslashReplaceErrors(PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
687 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
688 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
689 PyObject *restuple;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
690 PyObject *object;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
691 Py_ssize_t i;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
692 Py_ssize_t start;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
693 Py_ssize_t end;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
694 PyObject *res;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
695 unsigned char *outp;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
696 int ressize;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
697 Py_UCS4 c;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
698 if (PyUnicodeEncodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
699 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
700 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
701 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
702 if (!(object = PyUnicodeEncodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
703 return NULL;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
704 for (i = start, ressize = 0; i < end; ++i) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
705 /* object is guaranteed to be "ready" */
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
706 c = PyUnicode_READ_CHAR(object, i);
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
707 if (c >= 0x10000) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
708 ressize += 1+1+8;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
709 }
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
710 else if (c >= 0x100) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
711 ressize += 1+1+4;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
712 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
713 else
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
714 ressize += 1+1+2;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
715 }
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
716 res = PyUnicode_New(ressize, 127);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
717 if (res==NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
718 return NULL;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
719 for (i = start, outp = PyUnicode_1BYTE_DATA(res);
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
720 i < end; ++i) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
721 c = PyUnicode_READ_CHAR(object, i);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
722 *outp++ = '\\';
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
723 if (c >= 0x00010000) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
724 *outp++ = 'U';
72918
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
725 *outp++ = Py_hexdigits[(c>>28)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
726 *outp++ = Py_hexdigits[(c>>24)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
727 *outp++ = Py_hexdigits[(c>>20)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
728 *outp++ = Py_hexdigits[(c>>16)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
729 *outp++ = Py_hexdigits[(c>>12)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
730 *outp++ = Py_hexdigits[(c>>8)&0xf];
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
731 }
64590
fc76e8c617dc Issue #9804: ascii() now always represents unicode surrogate pairs as
Antoine Pitrou <solipsis@pitrou.net>
parents: 61466
diff changeset
732 else if (c >= 0x100) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
733 *outp++ = 'u';
72918
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
734 *outp++ = Py_hexdigits[(c>>12)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
735 *outp++ = Py_hexdigits[(c>>8)&0xf];
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
736 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
737 else
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
738 *outp++ = 'x';
72918
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
739 *outp++ = Py_hexdigits[(c>>4)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72848
diff changeset
740 *outp++ = Py_hexdigits[c&0xf];
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
741 }
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
742
76573
1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1)
Victor Stinner <victor.stinner@gmail.com>
parents: 74512
diff changeset
743 assert(_PyUnicode_CheckConsistency(res, 1));
1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1)
Victor Stinner <victor.stinner@gmail.com>
parents: 74512
diff changeset
744 restuple = Py_BuildValue("(Nn)", res, end);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
745 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
746 return restuple;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
747 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
748 else {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
749 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
750 return NULL;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
751 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
752 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
753
53687
52a89d723b6b Make PyCodec_SurrogateErrors static.
Martin v. Löwis <martin@v.loewis.de>
parents: 53681
diff changeset
754 /* This handler is declared static until someone demonstrates
52a89d723b6b Make PyCodec_SurrogateErrors static.
Martin v. Löwis <martin@v.loewis.de>
parents: 53681
diff changeset
755 a need to call it directly. */
52a89d723b6b Make PyCodec_SurrogateErrors static.
Martin v. Löwis <martin@v.loewis.de>
parents: 53681
diff changeset
756 static PyObject *
53965
fbcdae3e0ca1 Rename the surrogates error handler to surrogatepass.
Martin v. Löwis <martin@v.loewis.de>
parents: 53767
diff changeset
757 PyCodec_SurrogatePassErrors(PyObject *exc)
53681
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
758 {
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
759 PyObject *restuple;
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
760 PyObject *object;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
761 Py_ssize_t i;
53681
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
762 Py_ssize_t start;
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
763 Py_ssize_t end;
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
764 PyObject *res;
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
765 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
766 char *outp;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
767 if (PyUnicodeEncodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
768 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
769 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
770 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
771 if (!(object = PyUnicodeEncodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
772 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
773 res = PyBytes_FromStringAndSize(NULL, 3*(end-start));
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
774 if (!res) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
775 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
776 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
777 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
778 outp = PyBytes_AsString(res);
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
779 for (i = start; i < end; i++) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
780 /* object is guaranteed to be "ready" */
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
781 Py_UCS4 ch = PyUnicode_READ_CHAR(object, i);
80057
13dd8199c112 Issue #16330: Use surrogate-related macros
Victor Stinner <victor.stinner@gmail.com>
parents: 79954
diff changeset
782 if (!Py_UNICODE_IS_SURROGATE(ch)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
783 /* Not a surrogate, fail with original exception */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
784 PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
785 Py_DECREF(res);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
786 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
787 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
788 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
789 *outp++ = (char)(0xe0 | (ch >> 12));
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
790 *outp++ = (char)(0x80 | ((ch >> 6) & 0x3f));
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
791 *outp++ = (char)(0x80 | (ch & 0x3f));
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
792 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
793 restuple = Py_BuildValue("(On)", res, end);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
794 Py_DECREF(res);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
795 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
796 return restuple;
53681
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
797 }
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
798 else if (PyObject_IsInstance(exc, PyExc_UnicodeDecodeError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
799 unsigned char *p;
73359
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
800 Py_UCS4 ch = 0;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
801 if (PyUnicodeDecodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
802 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
803 if (!(object = PyUnicodeDecodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
804 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
805 if (!(p = (unsigned char*)PyBytes_AsString(object))) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
806 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
807 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
808 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
809 /* Try decoding a single surrogate character. If
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
810 there are more, let the codec call us again. */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
811 p += start;
80216
18ed608ee594 #16336: fix input checking in the surrogatepass error handler. Patch by Serhiy Storchaka.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 79953
diff changeset
812 if (PyBytes_GET_SIZE(object) - start >= 3 &&
18ed608ee594 #16336: fix input checking in the surrogatepass error handler. Patch by Serhiy Storchaka.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 79953
diff changeset
813 (p[0] & 0xf0) == 0xe0 &&
18ed608ee594 #16336: fix input checking in the surrogatepass error handler. Patch by Serhiy Storchaka.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 79953
diff changeset
814 (p[1] & 0xc0) == 0x80 &&
18ed608ee594 #16336: fix input checking in the surrogatepass error handler. Patch by Serhiy Storchaka.
Ezio Melotti <ezio.melotti@gmail.com>
parents: 79953
diff changeset
815 (p[2] & 0xc0) == 0x80) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
816 /* it's a three-byte code */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
817 ch = ((p[0] & 0x0f) << 12) + ((p[1] & 0x3f) << 6) + (p[2] & 0x3f);
80057
13dd8199c112 Issue #16330: Use surrogate-related macros
Victor Stinner <victor.stinner@gmail.com>
parents: 79954
diff changeset
818 if (!Py_UNICODE_IS_SURROGATE(ch))
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
819 /* it's not a surrogate - fail */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
820 ch = 0;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
821 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
822 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
823 if (ch == 0) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
824 PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
825 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
826 }
73359
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
827 res = PyUnicode_FromOrdinal(ch);
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
828 if (res == NULL)
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
829 return NULL;
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
830 return Py_BuildValue("(Nn)", res, start+3);
53681
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
831 }
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
832 else {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
833 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
834 return NULL;
53681
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
835 }
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
836 }
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
837
53767
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
838 static PyObject *
53966
7c29bfbfd352 Rename utf8b error handler to surrogateescape.
Martin v. Löwis <martin@v.loewis.de>
parents: 53965
diff changeset
839 PyCodec_SurrogateEscapeErrors(PyObject *exc)
53767
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
840 {
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
841 PyObject *restuple;
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
842 PyObject *object;
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
843 Py_ssize_t i;
53767
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
844 Py_ssize_t start;
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
845 Py_ssize_t end;
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
846 PyObject *res;
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
847 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
848 char *outp;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
849 if (PyUnicodeEncodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
850 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
851 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
852 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
853 if (!(object = PyUnicodeEncodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
854 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
855 res = PyBytes_FromStringAndSize(NULL, end-start);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
856 if (!res) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
857 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
858 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
859 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
860 outp = PyBytes_AsString(res);
73350
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
861 for (i = start; i < end; i++) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
862 /* object is guaranteed to be "ready" */
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing.
Martin v. Löwis <martin@v.loewis.de>
parents: 72925
diff changeset
863 Py_UCS4 ch = PyUnicode_READ_CHAR(object, i);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
864 if (ch < 0xdc80 || ch > 0xdcff) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
865 /* Not a UTF-8b surrogate, fail with original exception */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
866 PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
867 Py_DECREF(res);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
868 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
869 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
870 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
871 *outp++ = ch - 0xdc00;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
872 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
873 restuple = Py_BuildValue("(On)", res, end);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
874 Py_DECREF(res);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
875 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
876 return restuple;
53767
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
877 }
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
878 else if (PyObject_IsInstance(exc, PyExc_UnicodeDecodeError)) {
73359
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
879 PyObject *str;
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
880 unsigned char *p;
73359
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
881 Py_UCS2 ch[4]; /* decode up to 4 bad bytes. */
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
882 int consumed = 0;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
883 if (PyUnicodeDecodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
884 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
885 if (PyUnicodeDecodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
886 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
887 if (!(object = PyUnicodeDecodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
888 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
889 if (!(p = (unsigned char*)PyBytes_AsString(object))) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
890 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
891 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
892 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
893 while (consumed < 4 && consumed < end-start) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
894 /* Refuse to escape ASCII bytes. */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
895 if (p[start+consumed] < 128)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
896 break;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
897 ch[consumed] = 0xdc00 + p[start+consumed];
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
898 consumed++;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
899 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
900 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
901 if (!consumed) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
902 /* codec complained about ASCII byte. */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
903 PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
904 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
905 }
73359
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
906 str = PyUnicode_FromKindAndData(PyUnicode_2BYTE_KIND, ch, consumed);
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
907 if (str == NULL)
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
908 return NULL;
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c
Victor Stinner <victor.stinner@haypocalc.com>
parents: 73358
diff changeset
909 return Py_BuildValue("(Nn)", str, start+consumed);
53767
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
910 }
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
911 else {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
912 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
913 return NULL;
53767
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
914 }
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
915 }
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
916
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
917
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
918 static PyObject *strict_errors(PyObject *self, PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
919 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
920 return PyCodec_StrictErrors(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
921 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
922
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
923
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
924 static PyObject *ignore_errors(PyObject *self, PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
925 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
926 return PyCodec_IgnoreErrors(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
927 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
928
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
929
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
930 static PyObject *replace_errors(PyObject *self, PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
931 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
932 return PyCodec_ReplaceErrors(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
933 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
934
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
935
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
936 static PyObject *xmlcharrefreplace_errors(PyObject *self, PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
937 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
938 return PyCodec_XMLCharRefReplaceErrors(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
939 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
940
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
941
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
942 static PyObject *backslashreplace_errors(PyObject *self, PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
943 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
944 return PyCodec_BackslashReplaceErrors(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
945 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
946
53965
fbcdae3e0ca1 Rename the surrogates error handler to surrogatepass.
Martin v. Löwis <martin@v.loewis.de>
parents: 53767
diff changeset
947 static PyObject *surrogatepass_errors(PyObject *self, PyObject *exc)
53681
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
948 {
53965
fbcdae3e0ca1 Rename the surrogates error handler to surrogatepass.
Martin v. Löwis <martin@v.loewis.de>
parents: 53767
diff changeset
949 return PyCodec_SurrogatePassErrors(exc);
53681
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
950 }
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
Martin v. Löwis <martin@v.loewis.de>
parents: 49613
diff changeset
951
53966
7c29bfbfd352 Rename utf8b error handler to surrogateescape.
Martin v. Löwis <martin@v.loewis.de>
parents: 53965
diff changeset
952 static PyObject *surrogateescape_errors(PyObject *self, PyObject *exc)
53767
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
953 {
53966
7c29bfbfd352 Rename utf8b error handler to surrogateescape.
Martin v. Löwis <martin@v.loewis.de>
parents: 53965
diff changeset
954 return PyCodec_SurrogateEscapeErrors(exc);
53767
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
955 }
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in
Martin v. Löwis <martin@v.loewis.de>
parents: 53687
diff changeset
956
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
957 static int _PyCodecRegistry_Init(void)
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
958 {
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
959 static struct {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
960 char *name;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
961 PyMethodDef def;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
962 } methods[] =
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
963 {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
964 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
965 "strict",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
966 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
967 "strict_errors",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
968 strict_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
969 METH_O,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
970 PyDoc_STR("Implements the 'strict' error handling, which "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
971 "raises a UnicodeError on coding errors.")
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
972 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
973 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
974 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
975 "ignore",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
976 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
977 "ignore_errors",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
978 ignore_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
979 METH_O,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
980 PyDoc_STR("Implements the 'ignore' error handling, which "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
981 "ignores malformed data and continues.")
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
982 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
983 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
984 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
985 "replace",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
986 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
987 "replace_errors",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
988 replace_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
989 METH_O,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
990 PyDoc_STR("Implements the 'replace' error handling, which "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
991 "replaces malformed data with a replacement marker.")
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
992 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
993 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
994 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
995 "xmlcharrefreplace",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
996 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
997 "xmlcharrefreplace_errors",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
998 xmlcharrefreplace_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
999 METH_O,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1000 PyDoc_STR("Implements the 'xmlcharrefreplace' error handling, "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1001 "which replaces an unencodable character with the "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1002 "appropriate XML character reference.")
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1003 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1004 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1005 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1006 "backslashreplace",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1007 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1008 "backslashreplace_errors",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1009 backslashreplace_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1010 METH_O,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1011 PyDoc_STR("Implements the 'backslashreplace' error handling, "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1012 "which replaces an unencodable character with a "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1013 "backslashed escape sequence.")
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1014 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1015 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1016 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1017 "surrogatepass",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1018 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1019 "surrogatepass",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1020 surrogatepass_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1021 METH_O
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1022 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1023 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1024 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1025 "surrogateescape",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1026 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1027 "surrogateescape",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1028 surrogateescape_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1029 METH_O
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1030 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1031 }
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1032 };
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1033
31797
846b3ee7e106 Changed random calls to PyThreadState_Get() to use the macro
Nicholas Bastin <nick.bastin@gmail.com>
parents: 31273
diff changeset
1034 PyInterpreterState *interp = PyThreadState_GET()->interp;
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1035 PyObject *mod;
32533
201986d6d9d6 Fix a couple of signed/unsigned comparison warnings
Neal Norwitz <nnorwitz@gmail.com>
parents: 31797
diff changeset
1036 unsigned i;
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1037
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1038 if (interp->codec_search_path != NULL)
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1039 return 0;
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1040
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1041 interp->codec_search_path = PyList_New(0);
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1042 interp->codec_search_cache = PyDict_New();
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1043 interp->codec_error_registry = PyDict_New();
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1044
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1045 if (interp->codec_error_registry) {
72510
b534b56379cb Use the new Py_ARRAY_LENGTH macro
Victor Stinner <victor.stinner@haypocalc.com>
parents: 72475
diff changeset
1046 for (i = 0; i < Py_ARRAY_LENGTH(methods); ++i) {
81031
3a86a3f1d89a Issue #15422: get rid of PyCFunction_New macro
Andrew Svetlov <andrew.svetlov@gmail.com>
parents: 80218
diff changeset
1047 PyObject *func = PyCFunction_NewEx(&methods[i].def, NULL, NULL);
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1048 int res;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1049 if (!func)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1050 Py_FatalError("can't initialize codec error registry");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1051 res = PyCodec_RegisterError(methods[i].name, func);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1052 Py_DECREF(func);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1053 if (res)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1054 Py_FatalError("can't initialize codec error registry");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1055 }
25091
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401)
Walter Dörwald <walter@livinglogic.de>
parents: 24307
diff changeset
1056 }
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1057
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1058 if (interp->codec_search_path == NULL ||
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1059 interp->codec_search_cache == NULL ||
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1060 interp->codec_error_registry == NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1061 Py_FatalError("can't initialize codec registry");
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
1062
44442
fa2f7183b1af More PyImport_ImportModule -> PyImport_ImportModuleNoBlock
Christian Heimes <christian@cheimes.de>
parents: 44351
diff changeset
1063 mod = PyImport_ImportModuleNoBlock("encodings");
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1064 if (mod == NULL) {
61466
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from
Antoine Pitrou <solipsis@pitrou.net>
parents: 56894
diff changeset
1065 return -1;
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1066 }
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1067 Py_DECREF(mod);
49613
efef5fc45a9a Issue 3723: Fixed initialization of subinterpreters
Christian Heimes <christian@cheimes.de>
parents: 47496
diff changeset
1068 interp->codecs_initialized = 1;
28325
850626d47574 Fixed SF bug #663074. The codec system was using global static
Gustavo Niemeyer <gustavo@niemeyer.net>
parents: 27898
diff changeset
1069 return 0;
11972
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre
Guido van Rossum <guido@python.org>
parents:
diff changeset
1070 }