cpython-lru_cache-weakref: Python/codecs.c annotate

annotate Python/codecs.c @ 87338:d68df99d7a57

Issue #19619: Blacklist non-text codecs in method API str.encode, bytes.decode and bytearray.decode now use an internal API to throw LookupError for known non-text encodings, rather than attempting the encoding or decoding operation and then throwing a TypeError for an unexpected output type. The latter mechanism remains in place for third party non-text encodings.

author	Nick Coghlan <ncoghlan@gmail.com>
date	Fri, 22 Nov 2013 22:39:36 +1000
parents	0d9624f2ff43
children

rev	line source
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	1 /* ------------------------------------------------------------------------
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	2
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	3 Python Codec Registry and support functions
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	4
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	5 Written by Marc-Andre Lemburg (mal@lemburg.com).
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	6
13828 254f8b7c4a83 Changing the CNRI copyright notice according to CNRI's instructions. Guido van Rossum <guido@python.org> parents: 13658 diff changeset	7 Copyright (c) Corporation for National Research Initiatives.
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	8
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	9 ------------------------------------------------------------------------ */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	10
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	11 #include "Python.h"
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	12 #include <ctype.h>
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	13
72918 d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16 Victor Stinner <victor.stinner@haypocalc.com> parents: 72848 diff changeset	14 const char *Py_hexdigits = "0123456789abcdef";
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16 Victor Stinner <victor.stinner@haypocalc.com> parents: 72848 diff changeset	15
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	16 /* --- Codec Registry ----------------------------------------------------- */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	17
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	18 /* Import the standard encodings package which will register the first
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	19 codec search function.
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	20
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	21 This is done in a lazy way so that the Unicode implementation does
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	22 not downgrade startup time of scripts not needing it.
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	23
12195 bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12048 diff changeset	24 ImportErrors are silently ignored by this function. Only one try is
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12048 diff changeset	25 made.
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	26
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	27 */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	28
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	29 static int _PyCodecRegistry_Init(void); /* Forward */
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	30
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	31 int PyCodec_Register(PyObject *search_function)
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	32 {
31797 846b3ee7e106 Changed random calls to PyThreadState_Get() to use the macro Nicholas Bastin <nick.bastin@gmail.com> parents: 31273 diff changeset	33 PyInterpreterState *interp = PyThreadState_GET()->interp;
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	34 if (interp->codec_search_path == NULL && _PyCodecRegistry_Init())
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	35 goto onError;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	36 if (search_function == NULL) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	37 PyErr_BadArgument();
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	38 goto onError;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	39 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	40 if (!PyCallable_Check(search_function)) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	41 PyErr_SetString(PyExc_TypeError, "argument must be callable");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	42 goto onError;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	43 }
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	44 return PyList_Append(interp->codec_search_path, search_function);
12195 bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12048 diff changeset	45
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12048 diff changeset	46 onError:
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12048 diff changeset	47 return -1;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	48 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	49
12248 70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that Guido van Rossum <guido@python.org> parents: 12195 diff changeset	50 /* Convert a string to a normalized Python string: all characters are
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that Guido van Rossum <guido@python.org> parents: 12195 diff changeset	51 converted to lower case, spaces are replaced with underscores. */
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that Guido van Rossum <guido@python.org> parents: 12195 diff changeset	52
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	53 static
12248 70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that Guido van Rossum <guido@python.org> parents: 12195 diff changeset	54 PyObject normalizestring(const char string)
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	55 {
85157 e7f6cef7a4cc Issue #18722: Remove uses of the "register" keyword in C code. Antoine Pitrou <solipsis@pitrou.net> parents: 84571 diff changeset	56 size_t i;
13096 8c8de5b736dc Trent Mick's Win64 changes: size_t vs. int or long; also some overflow Guido van Rossum <guido@python.org> parents: 12648 diff changeset	57 size_t len = strlen(string);
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	58 char *p;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	59 PyObject *v;
43603 dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee: Guido van Rossum <guido@python.org> parents: 42534 diff changeset	60
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	61 if (len > PY_SSIZE_T_MAX) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	62 PyErr_SetString(PyExc_OverflowError, "string is too large");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	63 return NULL;
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	64 }
43603 dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee: Guido van Rossum <guido@python.org> parents: 42534 diff changeset	65
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee: Guido van Rossum <guido@python.org> parents: 42534 diff changeset	66 p = PyMem_Malloc(len + 1);
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee: Guido van Rossum <guido@python.org> parents: 42534 diff changeset	67 if (p == NULL)
84571 4975bcd67aa7 Issue #18408: normalizestring() now raises MemoryError on memory allocation failure Victor Stinner <victor.stinner@gmail.com> parents: 81031 diff changeset	68 return PyErr_NoMemory();
12248 70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that Guido van Rossum <guido@python.org> parents: 12195 diff changeset	69 for (i = 0; i < len; i++) {
85157 e7f6cef7a4cc Issue #18722: Remove uses of the "register" keyword in C code. Antoine Pitrou <solipsis@pitrou.net> parents: 84571 diff changeset	70 char ch = string[i];
12248 70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that Guido van Rossum <guido@python.org> parents: 12195 diff changeset	71 if (ch == ' ')
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that Guido van Rossum <guido@python.org> parents: 12195 diff changeset	72 ch = '-';
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that Guido van Rossum <guido@python.org> parents: 12195 diff changeset	73 else
71489 92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 64590 diff changeset	74 ch = Py_TOLOWER(Py_CHARMASK(ch));
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	75 p[i] = ch;
12248 70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that Guido van Rossum <guido@python.org> parents: 12195 diff changeset	76 }
43603 dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee: Guido van Rossum <guido@python.org> parents: 42534 diff changeset	77 p[i] = '\0';
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee: Guido van Rossum <guido@python.org> parents: 42534 diff changeset	78 v = PyUnicode_FromString(p);
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee: Guido van Rossum <guido@python.org> parents: 42534 diff changeset	79 if (v == NULL)
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee: Guido van Rossum <guido@python.org> parents: 42534 diff changeset	80 return NULL;
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee: Guido van Rossum <guido@python.org> parents: 42534 diff changeset	81 PyMem_Free(p);
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	82 return v;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	83 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	84
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	85 /* Lookup the given encoding and return a tuple providing the codec
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	86 facilities.
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	87
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	88 The encoding string is looked up converted to all lower-case
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	89 characters. This makes encodings looked up through this mechanism
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	90 effectively case-insensitive.
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	91
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	92 If no codec is found, a LookupError is set and NULL returned.
12195 bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12048 diff changeset	93
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12048 diff changeset	94 As side effect, this tries to load the encodings package, if not
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12048 diff changeset	95 yet done. This is part of the lazy load strategy for the encodings
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12048 diff changeset	96 package.
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12048 diff changeset	97
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12048 diff changeset	98 */
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	99
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	100 PyObject _PyCodec_Lookup(const char encoding)
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	101 {
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	102 PyInterpreterState *interp;
12048 f87b56eebacc Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12025 diff changeset	103 PyObject result, args = NULL, *v;
38453 29ae5e0fc348 Much-needed merge (using svnmerge.py this time) of trunk changes into p3yk. Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	104 Py_ssize_t i, len;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	105
12648 2a53f5dc0590 M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12248 diff changeset	106 if (encoding == NULL) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	107 PyErr_BadArgument();
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	108 goto onError;
12648 2a53f5dc0590 M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12248 diff changeset	109 }
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	110
31797 846b3ee7e106 Changed random calls to PyThreadState_Get() to use the macro Nicholas Bastin <nick.bastin@gmail.com> parents: 31273 diff changeset	111 interp = PyThreadState_GET()->interp;
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	112 if (interp->codec_search_path == NULL && _PyCodecRegistry_Init())
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	113 goto onError;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	114
12248 70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that Guido van Rossum <guido@python.org> parents: 12195 diff changeset	115 /* Convert the encoding to a normalized Python string: all
13620 5b727ab37092 Spelling fixes supplied by Rob W. W. Hooft. All these are fixes in either Thomas Wouters <thomas@python.org> parents: 13330 diff changeset	116 characters are converted to lower case, spaces and hyphens are
12248 70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that Guido van Rossum <guido@python.org> parents: 12195 diff changeset	117 replaced with underscores. */
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that Guido van Rossum <guido@python.org> parents: 12195 diff changeset	118 v = normalizestring(encoding);
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	119 if (v == NULL)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	120 goto onError;
43603 dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee: Guido van Rossum <guido@python.org> parents: 42534 diff changeset	121 PyUnicode_InternInPlace(&v);
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	122
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	123 /* First, try to lookup the name in the registry dictionary */
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	124 result = PyDict_GetItem(interp->codec_search_cache, v);
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	125 if (result != NULL) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	126 Py_INCREF(result);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	127 Py_DECREF(v);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	128 return result;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	129 }
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	130
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	131 /* Next, scan the search functions in order of registration */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	132 args = PyTuple_New(1);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	133 if (args == NULL)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	134 goto onError;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	135 PyTuple_SET_ITEM(args,0,v);
12048 f87b56eebacc Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12025 diff changeset	136
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	137 len = PyList_Size(interp->codec_search_path);
12048 f87b56eebacc Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12025 diff changeset	138 if (len < 0)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	139 goto onError;
12195 bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12048 diff changeset	140 if (len == 0) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	141 PyErr_SetString(PyExc_LookupError,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	142 "no codec search functions registered: "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	143 "can't find encoding");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	144 goto onError;
12195 bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12048 diff changeset	145 }
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	146
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	147 for (i = 0; i < len; i++) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	148 PyObject *func;
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	149
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	150 func = PyList_GetItem(interp->codec_search_path, i);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	151 if (func == NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	152 goto onError;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	153 result = PyEval_CallObject(func, args);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	154 if (result == NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	155 goto onError;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	156 if (result == Py_None) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	157 Py_DECREF(result);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	158 continue;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	159 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	160 if (!PyTuple_Check(result) \|\| PyTuple_GET_SIZE(result) != 4) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	161 PyErr_SetString(PyExc_TypeError,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	162 "codec search functions must return 4-tuples");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	163 Py_DECREF(result);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	164 goto onError;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	165 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	166 break;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	167 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	168 if (i == len) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	169 /* XXX Perhaps we should cache misses too ? */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	170 PyErr_Format(PyExc_LookupError,
25441 d32588c88633 Add encoding name in LookupError. Fixes #615013. Will backport to 2.2. Martin v. Löwis <martin@v.loewis.de> parents: 25095 diff changeset	171 "unknown encoding: %s", encoding);
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	172 goto onError;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	173 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	174
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	175 /* Cache and return the result */
42534 5f9e79d1e158 Handle error Neal Norwitz <nnorwitz@gmail.com> parents: 42057 diff changeset	176 if (PyDict_SetItem(interp->codec_search_cache, v, result) < 0) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	177 Py_DECREF(result);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	178 goto onError;
42534 5f9e79d1e158 Handle error Neal Norwitz <nnorwitz@gmail.com> parents: 42057 diff changeset	179 }
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	180 Py_DECREF(args);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	181 return result;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	182
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	183 onError:
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	184 Py_XDECREF(args);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	185 return NULL;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	186 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	187
47496 2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	188 /* Codec registry encoding check API. */
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	189
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	190 int PyCodec_KnownEncoding(const char *encoding)
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	191 {
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	192 PyObject *codecs;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	193
47496 2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	194 codecs = _PyCodec_Lookup(encoding);
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	195 if (!codecs) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	196 PyErr_Clear();
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	197 return 0;
47496 2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	198 }
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	199 else {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	200 Py_DECREF(codecs);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	201 return 1;
47496 2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	202 }
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	203 }
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	204
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	205 static
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	206 PyObject args_tuple(PyObject object,
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	207 const char *errors)
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	208 {
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	209 PyObject *args;
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	210
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	211 args = PyTuple_New(1 + (errors != NULL));
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	212 if (args == NULL)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	213 return NULL;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	214 Py_INCREF(object);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	215 PyTuple_SET_ITEM(args,0,object);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	216 if (errors) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	217 PyObject *v;
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	218
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	219 v = PyUnicode_FromString(errors);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	220 if (v == NULL) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	221 Py_DECREF(args);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	222 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	223 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	224 PyTuple_SET_ITEM(args, 1, v);
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	225 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	226 return args;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	227 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	228
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	229 /* Helper function to get a codec item */
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	230
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	231 static
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	232 PyObject codec_getitem(const char encoding, int index)
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	233 {
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	234 PyObject *codecs;
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	235 PyObject *v;
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	236
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	237 codecs = _PyCodec_Lookup(encoding);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	238 if (codecs == NULL)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	239 return NULL;
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	240 v = PyTuple_GET_ITEM(codecs, index);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	241 Py_DECREF(codecs);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	242 Py_INCREF(v);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	243 return v;
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	244 }
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	245
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	246 /* Helper function to create an incremental codec. */
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	247
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	248 static
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	249 PyObject codec_getincrementalcodec(const char encoding,
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	250 const char *errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	251 const char *attrname)
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	252 {
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	253 PyObject codecs, ret, *inccodec;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	254
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	255 codecs = _PyCodec_Lookup(encoding);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	256 if (codecs == NULL)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	257 return NULL;
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	258 inccodec = PyObject_GetAttrString(codecs, attrname);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	259 Py_DECREF(codecs);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	260 if (inccodec == NULL)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	261 return NULL;
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	262 if (errors)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	263 ret = PyObject_CallFunction(inccodec, "s", errors);
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	264 else
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	265 ret = PyObject_CallFunction(inccodec, NULL);
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	266 Py_DECREF(inccodec);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	267 return ret;
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	268 }
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	269
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	270 /* Helper function to create a stream codec. */
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	271
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	272 static
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	273 PyObject codec_getstreamcodec(const char encoding,
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	274 PyObject *stream,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	275 const char *errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	276 const int index)
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	277 {
39442 654c380cf8b9 Merged revisions 46753-51188 via svnmerge from Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	278 PyObject codecs, streamcodec, *codeccls;
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	279
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	280 codecs = _PyCodec_Lookup(encoding);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	281 if (codecs == NULL)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	282 return NULL;
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	283
39442 654c380cf8b9 Merged revisions 46753-51188 via svnmerge from Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	284 codeccls = PyTuple_GET_ITEM(codecs, index);
654c380cf8b9 Merged revisions 46753-51188 via svnmerge from Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	285 if (errors != NULL)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	286 streamcodec = PyObject_CallFunction(codeccls, "Os", stream, errors);
39442 654c380cf8b9 Merged revisions 46753-51188 via svnmerge from Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	287 else
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	288 streamcodec = PyObject_CallFunction(codeccls, "O", stream);
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	289 Py_DECREF(codecs);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	290 return streamcodec;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	291 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	292
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	293 /* Convenience APIs to query the Codec registry.
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	294
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	295 All APIs return a codec object with incremented refcount.
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	296
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	297 */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	298
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	299 PyObject PyCodec_Encoder(const char encoding)
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	300 {
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	301 return codec_getitem(encoding, 0);
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	302 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	303
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	304 PyObject PyCodec_Decoder(const char encoding)
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	305 {
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	306 return codec_getitem(encoding, 1);
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	307 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	308
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36751 diff changeset	309 PyObject PyCodec_IncrementalEncoder(const char encoding,
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	310 const char *errors)
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36751 diff changeset	311 {
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	312 return codec_getincrementalcodec(encoding, errors, "incrementalencoder");
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36751 diff changeset	313 }
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36751 diff changeset	314
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36751 diff changeset	315 PyObject PyCodec_IncrementalDecoder(const char encoding,
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	316 const char *errors)
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36751 diff changeset	317 {
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	318 return codec_getincrementalcodec(encoding, errors, "incrementaldecoder");
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36751 diff changeset	319 }
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36751 diff changeset	320
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	321 PyObject PyCodec_StreamReader(const char encoding,
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	322 PyObject *stream,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	323 const char *errors)
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	324 {
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	325 return codec_getstreamcodec(encoding, stream, errors, 2);
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	326 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	327
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	328 PyObject PyCodec_StreamWriter(const char encoding,
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	329 PyObject *stream,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	330 const char *errors)
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	331 {
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	332 return codec_getstreamcodec(encoding, stream, errors, 3);
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	333 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	334
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	335 /* Helper that tries to ensure the reported exception chain indicates the
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	336 * codec that was invoked to trigger the failure without changing the type
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	337 * of the exception raised.
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	338 */
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	339 static void
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	340 wrap_codec_error(const char *operation,
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	341 const char *encoding)
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	342 {
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	343 /* TrySetFromCause will replace the active exception with a suitably
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	344 * updated clone if it can, otherwise it will leave the original
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	345 * exception alone.
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	346 */
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	347 _PyErr_TrySetFromCause("%s with '%s' codec failed",
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	348 operation, encoding);
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	349 }
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	350
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	351 /* Encode an object (e.g. an Unicode object) using the given encoding
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	352 and return the resulting encoded object (usually a Python string).
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	353
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	354 errors is passed to the encoder factory as argument if non-NULL. */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	355
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	356 static PyObject *
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	357 _PyCodec_EncodeInternal(PyObject *object,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	358 PyObject *encoder,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	359 const char *encoding,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	360 const char *errors)
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	361 {
35939 79374eb6db51 Fix a few more ref leaks. Backport candidate Neal Norwitz <nnorwitz@gmail.com> parents: 32533 diff changeset	362 PyObject args = NULL, result = NULL;
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	363 PyObject *v = NULL;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	364
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	365 args = args_tuple(object, errors);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	366 if (args == NULL)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	367 goto onError;
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	368
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	369 result = PyEval_CallObject(encoder, args);
87109 4ea622c085ca Close 19609: narrow scope of codec exc chaining Nick Coghlan <ncoghlan@gmail.com> parents: 87084 diff changeset	370 if (result == NULL) {
4ea622c085ca Close 19609: narrow scope of codec exc chaining Nick Coghlan <ncoghlan@gmail.com> parents: 87084 diff changeset	371 wrap_codec_error("encoding", encoding);
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	372 goto onError;
87109 4ea622c085ca Close 19609: narrow scope of codec exc chaining Nick Coghlan <ncoghlan@gmail.com> parents: 87084 diff changeset	373 }
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	374
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	375 if (!PyTuple_Check(result) \|\|
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	376 PyTuple_GET_SIZE(result) != 2) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	377 PyErr_SetString(PyExc_TypeError,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	378 "encoder must return a tuple (object, integer)");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	379 goto onError;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	380 }
47496 2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	381 v = PyTuple_GET_ITEM(result,0);
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	382 Py_INCREF(v);
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	383 /* We don't check or use the second (integer) entry. */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	384
47496 2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	385 Py_DECREF(args);
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	386 Py_DECREF(encoder);
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	387 Py_DECREF(result);
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	388 return v;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	389
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	390 onError:
35939 79374eb6db51 Fix a few more ref leaks. Backport candidate Neal Norwitz <nnorwitz@gmail.com> parents: 32533 diff changeset	391 Py_XDECREF(result);
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	392 Py_XDECREF(args);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	393 Py_XDECREF(encoder);
47496 2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	394 return NULL;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	395 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	396
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	397 /* Decode an object (usually a Python string) using the given encoding
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	398 and return an equivalent object (e.g. an Unicode object).
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	399
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	400 errors is passed to the decoder factory as argument if non-NULL. */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	401
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	402 static PyObject *
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	403 _PyCodec_DecodeInternal(PyObject *object,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	404 PyObject *decoder,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	405 const char *encoding,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	406 const char *errors)
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	407 {
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	408 PyObject args = NULL, result = NULL;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	409 PyObject *v;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	410
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	411 args = args_tuple(object, errors);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	412 if (args == NULL)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	413 goto onError;
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	414
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	415 result = PyEval_CallObject(decoder,args);
87109 4ea622c085ca Close 19609: narrow scope of codec exc chaining Nick Coghlan <ncoghlan@gmail.com> parents: 87084 diff changeset	416 if (result == NULL) {
4ea622c085ca Close 19609: narrow scope of codec exc chaining Nick Coghlan <ncoghlan@gmail.com> parents: 87084 diff changeset	417 wrap_codec_error("decoding", encoding);
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	418 goto onError;
87109 4ea622c085ca Close 19609: narrow scope of codec exc chaining Nick Coghlan <ncoghlan@gmail.com> parents: 87084 diff changeset	419 }
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	420 if (!PyTuple_Check(result) \|\|
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	421 PyTuple_GET_SIZE(result) != 2) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	422 PyErr_SetString(PyExc_TypeError,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	423 "decoder must return a tuple (object,integer)");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	424 goto onError;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	425 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	426 v = PyTuple_GET_ITEM(result,0);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	427 Py_INCREF(v);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	428 /* We don't check or use the second (integer) entry. */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	429
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	430 Py_DECREF(args);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	431 Py_DECREF(decoder);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	432 Py_DECREF(result);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	433 return v;
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	434
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	435 onError:
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	436 Py_XDECREF(args);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	437 Py_XDECREF(decoder);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	438 Py_XDECREF(result);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	439 return NULL;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	440 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	441
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	442 /* Generic encoding/decoding API */
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	443 PyObject PyCodec_Encode(PyObject object,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	444 const char *encoding,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	445 const char *errors)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	446 {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	447 PyObject *encoder;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	448
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	449 encoder = PyCodec_Encoder(encoding);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	450 if (encoder == NULL)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	451 return NULL;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	452
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	453 return _PyCodec_EncodeInternal(object, encoder, encoding, errors);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	454 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	455
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	456 PyObject PyCodec_Decode(PyObject object,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	457 const char *encoding,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	458 const char *errors)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	459 {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	460 PyObject *decoder;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	461
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	462 decoder = PyCodec_Decoder(encoding);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	463 if (decoder == NULL)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	464 return NULL;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	465
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	466 return _PyCodec_DecodeInternal(object, decoder, encoding, errors);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	467 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	468
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	469 /* Text encoding/decoding API */
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	470 static
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	471 PyObject codec_getitem_checked(const char encoding,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	472 const char *operation_name,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	473 int index)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	474 {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	475 _Py_IDENTIFIER(_is_text_encoding);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	476 PyObject *codec;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	477 PyObject *attr;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	478 PyObject *v;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	479 int is_text_codec;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	480
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	481 codec = _PyCodec_Lookup(encoding);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	482 if (codec == NULL)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	483 return NULL;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	484
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	485 /* Backwards compatibility: assume any raw tuple describes a text
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	486 * encoding, and the same for anything lacking the private
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	487 * attribute.
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	488 */
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	489 if (!PyTuple_CheckExact(codec)) {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	490 attr = _PyObject_GetAttrId(codec, &PyId__is_text_encoding);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	491 if (attr == NULL) {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	492 if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	493 PyErr_Clear();
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	494 } else {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	495 Py_DECREF(codec);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	496 return NULL;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	497 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	498 } else {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	499 is_text_codec = PyObject_IsTrue(attr);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	500 Py_DECREF(attr);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	501 if (!is_text_codec) {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	502 Py_DECREF(codec);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	503 PyErr_Format(PyExc_LookupError,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	504 "'%.400s' is not a text encoding; "
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	505 "use codecs.%s() to handle arbitrary codecs",
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	506 encoding, operation_name);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	507 return NULL;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	508 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	509 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	510 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	511
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	512 v = PyTuple_GET_ITEM(codec, index);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	513 Py_DECREF(codec);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	514 Py_INCREF(v);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	515 return v;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	516 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	517
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	518 static PyObject * _PyCodec_TextEncoder(const char *encoding)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	519 {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	520 return codec_getitem_checked(encoding, "encode", 0);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	521 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	522
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	523 static PyObject * _PyCodec_TextDecoder(const char *encoding)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	524 {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	525 return codec_getitem_checked(encoding, "decode", 1);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	526 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	527
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	528 PyObject _PyCodec_EncodeText(PyObject object,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	529 const char *encoding,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	530 const char *errors)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	531 {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	532 PyObject *encoder;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	533
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	534 encoder = _PyCodec_TextEncoder(encoding);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	535 if (encoder == NULL)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	536 return NULL;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	537
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	538 return _PyCodec_EncodeInternal(object, encoder, encoding, errors);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	539 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	540
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	541 PyObject _PyCodec_DecodeText(PyObject object,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	542 const char *encoding,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	543 const char *errors)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	544 {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	545 PyObject *decoder;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	546
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	547 decoder = _PyCodec_TextDecoder(encoding);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	548 if (decoder == NULL)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	549 return NULL;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	550
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	551 return _PyCodec_DecodeInternal(object, decoder, encoding, errors);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	552 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	553
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	554 /* Register the error handling callback function error under the name
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	555 name. This function will be called by the codec when it encounters
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	556 an unencodable characters/undecodable bytes and doesn't know the
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	557 callback name, when name is specified as the error parameter
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	558 in the call to the encode/decode function.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	559 Return 0 on success, -1 on error */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	560 int PyCodec_RegisterError(const char name, PyObject error)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	561 {
31797 846b3ee7e106 Changed random calls to PyThreadState_Get() to use the macro Nicholas Bastin <nick.bastin@gmail.com> parents: 31273 diff changeset	562 PyInterpreterState *interp = PyThreadState_GET()->interp;
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	563 if (interp->codec_search_path == NULL && _PyCodecRegistry_Init())
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	564 return -1;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	565 if (!PyCallable_Check(error)) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	566 PyErr_SetString(PyExc_TypeError, "handler must be callable");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	567 return -1;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	568 }
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	569 return PyDict_SetItemString(interp->codec_error_registry,
86484 3055e61f1e66 Issue #1772673: The type of `char` arguments now changed to `const char`. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	570 name, error);
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	571 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	572
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	573 /* Lookup the error handling callback function registered under the
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	574 name error. As a special case NULL can be passed, in which case
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	575 the error handling callback for strict encoding will be returned. */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	576 PyObject PyCodec_LookupError(const char name)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	577 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	578 PyObject *handler = NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	579
31797 846b3ee7e106 Changed random calls to PyThreadState_Get() to use the macro Nicholas Bastin <nick.bastin@gmail.com> parents: 31273 diff changeset	580 PyInterpreterState *interp = PyThreadState_GET()->interp;
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	581 if (interp->codec_search_path == NULL && _PyCodecRegistry_Init())
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	582 return NULL;
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	583
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	584 if (name==NULL)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	585 name = "strict";
86484 3055e61f1e66 Issue #1772673: The type of `char` arguments now changed to `const char`. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	586 handler = PyDict_GetItemString(interp->codec_error_registry, name);
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	587 if (!handler)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	588 PyErr_Format(PyExc_LookupError, "unknown error handler name '%.400s'", name);
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	589 else
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	590 Py_INCREF(handler);
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	591 return handler;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	592 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	593
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	594 static void wrong_exception_type(PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	595 {
72925 7109f31300fb Rename _Py_identifier to _Py_IDENTIFIER. Martin v. Löwis <martin@v.loewis.de> parents: 72918 diff changeset	596 _Py_IDENTIFIER(__class__);
7109f31300fb Rename _Py_identifier to _Py_IDENTIFIER. Martin v. Löwis <martin@v.loewis.de> parents: 72918 diff changeset	597 _Py_IDENTIFIER(__name__);
72848 81380082d216 Use identifier API for PyObject_GetAttrString. Martin v. Löwis <martin@v.loewis.de> parents: 72591 diff changeset	598 PyObject *type = _PyObject_GetAttrId(exc, &PyId___class__);
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	599 if (type != NULL) {
72848 81380082d216 Use identifier API for PyObject_GetAttrString. Martin v. Löwis <martin@v.loewis.de> parents: 72591 diff changeset	600 PyObject *name = _PyObject_GetAttrId(type, &PyId___name__);
41878 089f449a20cd Change PyErr_Format() to generate a unicode string (by using Walter Dörwald <walter@livinglogic.de> parents: 41587 diff changeset	601 Py_DECREF(type);
089f449a20cd Change PyErr_Format() to generate a unicode string (by using Walter Dörwald <walter@livinglogic.de> parents: 41587 diff changeset	602 if (name != NULL) {
089f449a20cd Change PyErr_Format() to generate a unicode string (by using Walter Dörwald <walter@livinglogic.de> parents: 41587 diff changeset	603 PyErr_Format(PyExc_TypeError,
089f449a20cd Change PyErr_Format() to generate a unicode string (by using Walter Dörwald <walter@livinglogic.de> parents: 41587 diff changeset	604 "don't know how to handle %S in error callback", name);
089f449a20cd Change PyErr_Format() to generate a unicode string (by using Walter Dörwald <walter@livinglogic.de> parents: 41587 diff changeset	605 Py_DECREF(name);
089f449a20cd Change PyErr_Format() to generate a unicode string (by using Walter Dörwald <walter@livinglogic.de> parents: 41587 diff changeset	606 }
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	607 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	608 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	609
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	610 PyObject PyCodec_StrictErrors(PyObject exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	611 {
36751 8e04de2480e4 PEP 352 implementation. Creates a new base class, BaseException, which has an Brett Cannon <bcannon@gmail.com> parents: 36712 diff changeset	612 if (PyExceptionInstance_Check(exc))
8e04de2480e4 PEP 352 implementation. Creates a new base class, BaseException, which has an Brett Cannon <bcannon@gmail.com> parents: 36712 diff changeset	613 PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	614 else
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	615 PyErr_SetString(PyExc_TypeError, "codec must pass exception instance");
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	616 return NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	617 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	618
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	619
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	620 PyObject PyCodec_IgnoreErrors(PyObject exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	621 {
36501 7fdc639bc5b4 Merge ssize_t branch. Martin v. Löwis <martin@v.loewis.de> parents: 35939 diff changeset	622 Py_ssize_t end;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	623 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	624 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	625 return NULL;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	626 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	627 else if (PyObject_IsInstance(exc, PyExc_UnicodeDecodeError)) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	628 if (PyUnicodeDecodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	629 return NULL;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	630 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	631 else if (PyObject_IsInstance(exc, PyExc_UnicodeTranslateError)) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	632 if (PyUnicodeTranslateError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	633 return NULL;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	634 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	635 else {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	636 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	637 return NULL;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	638 }
73800 6407294eb3c6 PyCodec_IgnoreErrors() avoids the deprecated "u#" format Victor Stinner <victor.stinner@haypocalc.com> parents: 73359 diff changeset	639 return Py_BuildValue("(Nn)", PyUnicode_New(0, 0), end);
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	640 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	641
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	642
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	643 PyObject PyCodec_ReplaceErrors(PyObject exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	644 {
72475 8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	645 Py_ssize_t start, end, i, len;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	646
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	647 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	648 PyObject *res;
72475 8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	649 int kind;
8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	650 void *data;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	651 if (PyUnicodeEncodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	652 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	653 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	654 return NULL;
72475 8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	655 len = end - start;
8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	656 res = PyUnicode_New(len, '?');
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	657 if (res == NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	658 return NULL;
72475 8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	659 kind = PyUnicode_KIND(res);
8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	660 data = PyUnicode_DATA(res);
8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	661 for (i = 0; i < len; ++i)
8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	662 PyUnicode_WRITE(kind, data, i, '?');
76573 1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1) Victor Stinner <victor.stinner@gmail.com> parents: 74512 diff changeset	663 assert(_PyUnicode_CheckConsistency(res, 1));
72475 8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	664 return Py_BuildValue("(Nn)", res, end);
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	665 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	666 else if (PyObject_IsInstance(exc, PyExc_UnicodeDecodeError)) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	667 if (PyUnicodeDecodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	668 return NULL;
72591 611c57aa694a PyCodec_ReplaceErrors() uses "C" format instead of "u#" to build result Victor Stinner <victor.stinner@haypocalc.com> parents: 72510 diff changeset	669 return Py_BuildValue("(Cn)",
611c57aa694a PyCodec_ReplaceErrors() uses "C" format instead of "u#" to build result Victor Stinner <victor.stinner@haypocalc.com> parents: 72510 diff changeset	670 (int)Py_UNICODE_REPLACEMENT_CHARACTER,
611c57aa694a PyCodec_ReplaceErrors() uses "C" format instead of "u#" to build result Victor Stinner <victor.stinner@haypocalc.com> parents: 72510 diff changeset	671 end);
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	672 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	673 else if (PyObject_IsInstance(exc, PyExc_UnicodeTranslateError)) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	674 PyObject *res;
72475 8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	675 int kind;
8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	676 void *data;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	677 if (PyUnicodeTranslateError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	678 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	679 if (PyUnicodeTranslateError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	680 return NULL;
72475 8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	681 len = end - start;
8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	682 res = PyUnicode_New(len, Py_UNICODE_REPLACEMENT_CHARACTER);
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	683 if (res == NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	684 return NULL;
72475 8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	685 kind = PyUnicode_KIND(res);
8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	686 data = PyUnicode_DATA(res);
8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	687 for (i=0; i < len; i++)
8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	688 PyUnicode_WRITE(kind, data, i, Py_UNICODE_REPLACEMENT_CHARACTER);
76573 1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1) Victor Stinner <victor.stinner@gmail.com> parents: 74512 diff changeset	689 assert(_PyUnicode_CheckConsistency(res, 1));
72475 8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	690 return Py_BuildValue("(Nn)", res, end);
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	691 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	692 else {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	693 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	694 return NULL;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	695 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	696 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	697
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	698 PyObject PyCodec_XMLCharRefReplaceErrors(PyObject exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	699 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	700 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	701 PyObject *restuple;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	702 PyObject *object;
73358 2df7d749ee3a PyCodec_XMLCharRefReplaceError(): Remove unused variable Victor Stinner <victor.stinner@haypocalc.com> parents: 73351 diff changeset	703 Py_ssize_t i;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	704 Py_ssize_t start;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	705 Py_ssize_t end;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	706 PyObject *res;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	707 unsigned char *outp;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	708 int ressize;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	709 Py_UCS4 ch;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	710 if (PyUnicodeEncodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	711 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	712 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	713 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	714 if (!(object = PyUnicodeEncodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	715 return NULL;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	716 for (i = start, ressize = 0; i < end; ++i) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	717 /* object is guaranteed to be "ready" */
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	718 ch = PyUnicode_READ_CHAR(object, i);
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	719 if (ch<10)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	720 ressize += 2+1+1;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	721 else if (ch<100)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	722 ressize += 2+2+1;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	723 else if (ch<1000)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	724 ressize += 2+3+1;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	725 else if (ch<10000)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	726 ressize += 2+4+1;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	727 else if (ch<100000)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	728 ressize += 2+5+1;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	729 else if (ch<1000000)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	730 ressize += 2+6+1;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	731 else
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	732 ressize += 2+7+1;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	733 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	734 /* allocate replacement */
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	735 res = PyUnicode_New(ressize, 127);
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	736 if (res == NULL) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	737 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	738 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	739 }
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	740 outp = PyUnicode_1BYTE_DATA(res);
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	741 /* generate replacement */
73358 2df7d749ee3a PyCodec_XMLCharRefReplaceError(): Remove unused variable Victor Stinner <victor.stinner@haypocalc.com> parents: 73351 diff changeset	742 for (i = start; i < end; ++i) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	743 int digits;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	744 int base;
73351 2bec7c452b39 Fix C89 incompatibility. Martin v. Löwis <martin@v.loewis.de> parents: 73350 diff changeset	745 ch = PyUnicode_READ_CHAR(object, i);
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	746 *outp++ = '&';
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	747 *outp++ = '#';
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	748 if (ch<10) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	749 digits = 1;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	750 base = 1;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	751 }
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	752 else if (ch<100) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	753 digits = 2;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	754 base = 10;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	755 }
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	756 else if (ch<1000) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	757 digits = 3;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	758 base = 100;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	759 }
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	760 else if (ch<10000) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	761 digits = 4;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	762 base = 1000;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	763 }
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	764 else if (ch<100000) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	765 digits = 5;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	766 base = 10000;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	767 }
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	768 else if (ch<1000000) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	769 digits = 6;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	770 base = 100000;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	771 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	772 else {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	773 digits = 7;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	774 base = 1000000;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	775 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	776 while (digits-->0) {
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	777 *outp++ = '0' + ch/base;
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	778 ch %= base;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	779 base /= 10;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	780 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	781 *outp++ = ';';
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	782 }
76573 1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1) Victor Stinner <victor.stinner@gmail.com> parents: 74512 diff changeset	783 assert(_PyUnicode_CheckConsistency(res, 1));
1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1) Victor Stinner <victor.stinner@gmail.com> parents: 74512 diff changeset	784 restuple = Py_BuildValue("(Nn)", res, end);
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	785 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	786 return restuple;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	787 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	788 else {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	789 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	790 return NULL;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	791 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	792 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	793
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	794 PyObject PyCodec_BackslashReplaceErrors(PyObject exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	795 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	796 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	797 PyObject *restuple;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	798 PyObject *object;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	799 Py_ssize_t i;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	800 Py_ssize_t start;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	801 Py_ssize_t end;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	802 PyObject *res;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	803 unsigned char *outp;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	804 int ressize;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	805 Py_UCS4 c;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	806 if (PyUnicodeEncodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	807 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	808 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	809 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	810 if (!(object = PyUnicodeEncodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	811 return NULL;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	812 for (i = start, ressize = 0; i < end; ++i) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	813 /* object is guaranteed to be "ready" */
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	814 c = PyUnicode_READ_CHAR(object, i);
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	815 if (c >= 0x10000) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	816 ressize += 1+1+8;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	817 }
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	818 else if (c >= 0x100) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	819 ressize += 1+1+4;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	820 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	821 else
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	822 ressize += 1+1+2;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	823 }
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	824 res = PyUnicode_New(ressize, 127);
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	825 if (res==NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	826 return NULL;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	827 for (i = start, outp = PyUnicode_1BYTE_DATA(res);
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	828 i < end; ++i) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	829 c = PyUnicode_READ_CHAR(object, i);
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	830 *outp++ = '\\';
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	831 if (c >= 0x00010000) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	832 *outp++ = 'U';
72918 d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16 Victor Stinner <victor.stinner@haypocalc.com> parents: 72848 diff changeset	833 *outp++ = Py_hexdigits[(c>>28)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16 Victor Stinner <victor.stinner@haypocalc.com> parents: 72848 diff changeset	834 *outp++ = Py_hexdigits[(c>>24)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16 Victor Stinner <victor.stinner@haypocalc.com> parents: 72848 diff changeset	835 *outp++ = Py_hexdigits[(c>>20)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16 Victor Stinner <victor.stinner@haypocalc.com> parents: 72848 diff changeset	836 *outp++ = Py_hexdigits[(c>>16)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16 Victor Stinner <victor.stinner@haypocalc.com> parents: 72848 diff changeset	837 *outp++ = Py_hexdigits[(c>>12)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16 Victor Stinner <victor.stinner@haypocalc.com> parents: 72848 diff changeset	838 *outp++ = Py_hexdigits[(c>>8)&0xf];
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	839 }
64590 fc76e8c617dc Issue #9804: ascii() now always represents unicode surrogate pairs as Antoine Pitrou <solipsis@pitrou.net> parents: 61466 diff changeset	840 else if (c >= 0x100) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	841 *outp++ = 'u';
72918 d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16 Victor Stinner <victor.stinner@haypocalc.com> parents: 72848 diff changeset	842 *outp++ = Py_hexdigits[(c>>12)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16 Victor Stinner <victor.stinner@haypocalc.com> parents: 72848 diff changeset	843 *outp++ = Py_hexdigits[(c>>8)&0xf];
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	844 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	845 else
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	846 *outp++ = 'x';
72918 d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16 Victor Stinner <victor.stinner@haypocalc.com> parents: 72848 diff changeset	847 *outp++ = Py_hexdigits[(c>>4)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16 Victor Stinner <victor.stinner@haypocalc.com> parents: 72848 diff changeset	848 *outp++ = Py_hexdigits[c&0xf];
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	849 }
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	850
76573 1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1) Victor Stinner <victor.stinner@gmail.com> parents: 74512 diff changeset	851 assert(_PyUnicode_CheckConsistency(res, 1));
1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1) Victor Stinner <victor.stinner@gmail.com> parents: 74512 diff changeset	852 restuple = Py_BuildValue("(Nn)", res, end);
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	853 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	854 return restuple;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	855 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	856 else {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	857 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	858 return NULL;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	859 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	860 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	861
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	862 #define ENC_UTF8 0
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	863 #define ENC_UTF16BE 1
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	864 #define ENC_UTF16LE 2
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	865 #define ENC_UTF32BE 3
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	866 #define ENC_UTF32LE 4
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	867
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	868 static int
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	869 get_standard_encoding(const char encoding, int bytelength)
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	870 {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	871 if (Py_TOLOWER(encoding[0]) == 'u' &&
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	872 Py_TOLOWER(encoding[1]) == 't' &&
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	873 Py_TOLOWER(encoding[2]) == 'f') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	874 encoding += 3;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	875 if (encoding == '-' \|\| encoding == '_' )
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	876 encoding++;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	877 if (encoding[0] == '1' && encoding[1] == '6') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	878 encoding += 2;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	879 *bytelength = 2;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	880 if (*encoding == '\0') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	881 #ifdef WORDS_BIGENDIAN
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	882 return ENC_UTF16BE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	883 #else
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	884 return ENC_UTF16LE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	885 #endif
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	886 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	887 if (encoding == '-' \|\| encoding == '_' )
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	888 encoding++;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	889 if (Py_TOLOWER(encoding[1]) == 'e' && encoding[2] == '\0') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	890 if (Py_TOLOWER(encoding[0]) == 'b')
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	891 return ENC_UTF16BE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	892 if (Py_TOLOWER(encoding[0]) == 'l')
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	893 return ENC_UTF16LE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	894 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	895 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	896 else if (encoding[0] == '3' && encoding[1] == '2') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	897 encoding += 2;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	898 *bytelength = 4;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	899 if (*encoding == '\0') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	900 #ifdef WORDS_BIGENDIAN
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	901 return ENC_UTF32BE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	902 #else
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	903 return ENC_UTF32LE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	904 #endif
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	905 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	906 if (encoding == '-' \|\| encoding == '_' )
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	907 encoding++;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	908 if (Py_TOLOWER(encoding[1]) == 'e' && encoding[2] == '\0') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	909 if (Py_TOLOWER(encoding[0]) == 'b')
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	910 return ENC_UTF32BE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	911 if (Py_TOLOWER(encoding[0]) == 'l')
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	912 return ENC_UTF32LE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	913 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	914 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	915 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	916 /* utf-8 */
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	917 *bytelength = 3;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	918 return ENC_UTF8;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	919 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	920
53687 52a89d723b6b Make PyCodec_SurrogateErrors static. Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	921 /* This handler is declared static until someone demonstrates
52a89d723b6b Make PyCodec_SurrogateErrors static. Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	922 a need to call it directly. */
52a89d723b6b Make PyCodec_SurrogateErrors static. Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	923 static PyObject *
53965 fbcdae3e0ca1 Rename the surrogates error handler to surrogatepass. Martin v. Löwis <martin@v.loewis.de> parents: 53767 diff changeset	924 PyCodec_SurrogatePassErrors(PyObject *exc)
53681 52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	925 {
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	926 PyObject *restuple;
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	927 PyObject *object;
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	928 PyObject *encode;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	929 char *encoding;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	930 int code;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	931 int bytelength;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	932 Py_ssize_t i;
53681 52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	933 Py_ssize_t start;
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	934 Py_ssize_t end;
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	935 PyObject *res;
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	936 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	937 unsigned char *outp;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	938 if (PyUnicodeEncodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	939 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	940 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	941 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	942 if (!(object = PyUnicodeEncodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	943 return NULL;
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	944 if (!(encode = PyUnicodeEncodeError_GetEncoding(exc))) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	945 Py_DECREF(object);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	946 return NULL;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	947 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	948 if (!(encoding = PyUnicode_AsUTF8(encode))) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	949 Py_DECREF(object);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	950 Py_DECREF(encode);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	951 return NULL;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	952 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	953 code = get_standard_encoding(encoding, &bytelength);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	954 Py_DECREF(encode);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	955
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	956 res = PyBytes_FromStringAndSize(NULL, bytelength*(end-start));
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	957 if (!res) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	958 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	959 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	960 }
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	961 outp = (unsigned char*)PyBytes_AsString(res);
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	962 for (i = start; i < end; i++) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	963 /* object is guaranteed to be "ready" */
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	964 Py_UCS4 ch = PyUnicode_READ_CHAR(object, i);
80057 13dd8199c112 Issue #16330: Use surrogate-related macros Victor Stinner <victor.stinner@gmail.com> parents: 79954 diff changeset	965 if (!Py_UNICODE_IS_SURROGATE(ch)) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	966 /* Not a surrogate, fail with original exception */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	967 PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	968 Py_DECREF(res);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	969 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	970 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	971 }
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	972 switch (code) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	973 case ENC_UTF8:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	974 *outp++ = (unsigned char)(0xe0 \| (ch >> 12));
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	975 *outp++ = (unsigned char)(0x80 \| ((ch >> 6) & 0x3f));
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	976 *outp++ = (unsigned char)(0x80 \| (ch & 0x3f));
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	977 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	978 case ENC_UTF16LE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	979 *outp++ = (unsigned char) ch;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	980 *outp++ = (unsigned char)(ch >> 8);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	981 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	982 case ENC_UTF16BE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	983 *outp++ = (unsigned char)(ch >> 8);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	984 *outp++ = (unsigned char) ch;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	985 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	986 case ENC_UTF32LE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	987 *outp++ = (unsigned char) ch;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	988 *outp++ = (unsigned char)(ch >> 8);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	989 *outp++ = (unsigned char)(ch >> 16);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	990 *outp++ = (unsigned char)(ch >> 24);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	991 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	992 case ENC_UTF32BE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	993 *outp++ = (unsigned char)(ch >> 24);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	994 *outp++ = (unsigned char)(ch >> 16);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	995 *outp++ = (unsigned char)(ch >> 8);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	996 *outp++ = (unsigned char) ch;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	997 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	998 }
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	999 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1000 restuple = Py_BuildValue("(On)", res, end);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1001 Py_DECREF(res);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1002 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1003 return restuple;
53681 52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	1004 }
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	1005 else if (PyObject_IsInstance(exc, PyExc_UnicodeDecodeError)) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1006 unsigned char *p;
73359 e3b6458056c7 Avoid the Py_UNICODE type in codecs.c Victor Stinner <victor.stinner@haypocalc.com> parents: 73358 diff changeset	1007 Py_UCS4 ch = 0;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1008 if (PyUnicodeDecodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1009 return NULL;
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1010 if (PyUnicodeDecodeError_GetEnd(exc, &end))
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1011 return NULL;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1012 if (!(object = PyUnicodeDecodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1013 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1014 if (!(p = (unsigned char*)PyBytes_AsString(object))) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1015 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1016 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1017 }
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1018 if (!(encode = PyUnicodeDecodeError_GetEncoding(exc))) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1019 Py_DECREF(object);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1020 return NULL;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1021 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1022 if (!(encoding = PyUnicode_AsUTF8(encode))) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1023 Py_DECREF(object);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1024 Py_DECREF(encode);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1025 return NULL;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1026 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1027 code = get_standard_encoding(encoding, &bytelength);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1028 Py_DECREF(encode);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1029
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1030 /* Try decoding a single surrogate character. If
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1031 there are more, let the codec call us again. */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1032 p += start;
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1033 if (PyBytes_GET_SIZE(object) - start >= bytelength) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1034 switch (code) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1035 case ENC_UTF8:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1036 if ((p[0] & 0xf0) == 0xe0 &&
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1037 (p[1] & 0xc0) == 0x80 &&
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1038 (p[2] & 0xc0) == 0x80) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1039 /* it's a three-byte code */
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1040 ch = ((p[0] & 0x0f) << 12) + ((p[1] & 0x3f) << 6) + (p[2] & 0x3f);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1041 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1042 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1043 case ENC_UTF16LE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1044 ch = p[1] << 8 \| p[0];
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1045 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1046 case ENC_UTF16BE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1047 ch = p[0] << 8 \| p[1];
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1048 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1049 case ENC_UTF32LE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1050 ch = (p[3] << 24) \| (p[2] << 16) \| (p[1] << 8) \| p[0];
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1051 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1052 case ENC_UTF32BE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1053 ch = (p[0] << 24) \| (p[1] << 16) \| (p[2] << 8) \| p[3];
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1054 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1055 }
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1056 }
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1057
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1058 Py_DECREF(object);
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1059 if (!Py_UNICODE_IS_SURROGATE(ch)) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1060 /* it's not a surrogate - fail */
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1061 PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1062 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1063 }
73359 e3b6458056c7 Avoid the Py_UNICODE type in codecs.c Victor Stinner <victor.stinner@haypocalc.com> parents: 73358 diff changeset	1064 res = PyUnicode_FromOrdinal(ch);
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c Victor Stinner <victor.stinner@haypocalc.com> parents: 73358 diff changeset	1065 if (res == NULL)
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c Victor Stinner <victor.stinner@haypocalc.com> parents: 73358 diff changeset	1066 return NULL;
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1067 return Py_BuildValue("(Nn)", res, start + bytelength);
53681 52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	1068 }
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	1069 else {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1070 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1071 return NULL;
53681 52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	1072 }
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	1073 }
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	1074
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1075 static PyObject *
53966 7c29bfbfd352 Rename utf8b error handler to surrogateescape. Martin v. Löwis <martin@v.loewis.de> parents: 53965 diff changeset	1076 PyCodec_SurrogateEscapeErrors(PyObject *exc)
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1077 {
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1078 PyObject *restuple;
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1079 PyObject *object;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	1080 Py_ssize_t i;
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1081 Py_ssize_t start;
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1082 Py_ssize_t end;
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1083 PyObject *res;
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1084 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1085 char *outp;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1086 if (PyUnicodeEncodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1087 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1088 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1089 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1090 if (!(object = PyUnicodeEncodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1091 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1092 res = PyBytes_FromStringAndSize(NULL, end-start);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1093 if (!res) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1094 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1095 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1096 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1097 outp = PyBytes_AsString(res);
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	1098 for (i = start; i < end; i++) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	1099 /* object is guaranteed to be "ready" */
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	1100 Py_UCS4 ch = PyUnicode_READ_CHAR(object, i);
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1101 if (ch < 0xdc80 \|\| ch > 0xdcff) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1102 /* Not a UTF-8b surrogate, fail with original exception */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1103 PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1104 Py_DECREF(res);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1105 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1106 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1107 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1108 *outp++ = ch - 0xdc00;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1109 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1110 restuple = Py_BuildValue("(On)", res, end);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1111 Py_DECREF(res);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1112 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1113 return restuple;
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1114 }
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1115 else if (PyObject_IsInstance(exc, PyExc_UnicodeDecodeError)) {
73359 e3b6458056c7 Avoid the Py_UNICODE type in codecs.c Victor Stinner <victor.stinner@haypocalc.com> parents: 73358 diff changeset	1116 PyObject *str;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1117 unsigned char *p;
73359 e3b6458056c7 Avoid the Py_UNICODE type in codecs.c Victor Stinner <victor.stinner@haypocalc.com> parents: 73358 diff changeset	1118 Py_UCS2 ch[4]; /* decode up to 4 bad bytes. */
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1119 int consumed = 0;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1120 if (PyUnicodeDecodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1121 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1122 if (PyUnicodeDecodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1123 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1124 if (!(object = PyUnicodeDecodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1125 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1126 if (!(p = (unsigned char*)PyBytes_AsString(object))) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1127 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1128 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1129 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1130 while (consumed < 4 && consumed < end-start) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1131 /* Refuse to escape ASCII bytes. */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1132 if (p[start+consumed] < 128)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1133 break;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1134 ch[consumed] = 0xdc00 + p[start+consumed];
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1135 consumed++;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1136 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1137 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1138 if (!consumed) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1139 /* codec complained about ASCII byte. */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1140 PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1141 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1142 }
73359 e3b6458056c7 Avoid the Py_UNICODE type in codecs.c Victor Stinner <victor.stinner@haypocalc.com> parents: 73358 diff changeset	1143 str = PyUnicode_FromKindAndData(PyUnicode_2BYTE_KIND, ch, consumed);
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c Victor Stinner <victor.stinner@haypocalc.com> parents: 73358 diff changeset	1144 if (str == NULL)
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c Victor Stinner <victor.stinner@haypocalc.com> parents: 73358 diff changeset	1145 return NULL;
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c Victor Stinner <victor.stinner@haypocalc.com> parents: 73358 diff changeset	1146 return Py_BuildValue("(Nn)", str, start+consumed);
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1147 }
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1148 else {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1149 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1150 return NULL;
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1151 }
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1152 }
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1153
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1154
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1155 static PyObject strict_errors(PyObject self, PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1156 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1157 return PyCodec_StrictErrors(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1158 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1159
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1160
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1161 static PyObject ignore_errors(PyObject self, PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1162 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1163 return PyCodec_IgnoreErrors(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1164 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1165
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1166
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1167 static PyObject replace_errors(PyObject self, PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1168 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1169 return PyCodec_ReplaceErrors(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1170 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1171
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1172
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1173 static PyObject xmlcharrefreplace_errors(PyObject self, PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1174 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1175 return PyCodec_XMLCharRefReplaceErrors(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1176 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1177
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1178
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1179 static PyObject backslashreplace_errors(PyObject self, PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1180 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1181 return PyCodec_BackslashReplaceErrors(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1182 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1183
53965 fbcdae3e0ca1 Rename the surrogates error handler to surrogatepass. Martin v. Löwis <martin@v.loewis.de> parents: 53767 diff changeset	1184 static PyObject surrogatepass_errors(PyObject self, PyObject *exc)
53681 52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	1185 {
53965 fbcdae3e0ca1 Rename the surrogates error handler to surrogatepass. Martin v. Löwis <martin@v.loewis.de> parents: 53767 diff changeset	1186 return PyCodec_SurrogatePassErrors(exc);
53681 52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	1187 }
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	1188
53966 7c29bfbfd352 Rename utf8b error handler to surrogateescape. Martin v. Löwis <martin@v.loewis.de> parents: 53965 diff changeset	1189 static PyObject surrogateescape_errors(PyObject self, PyObject *exc)
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1190 {
53966 7c29bfbfd352 Rename utf8b error handler to surrogateescape. Martin v. Löwis <martin@v.loewis.de> parents: 53965 diff changeset	1191 return PyCodec_SurrogateEscapeErrors(exc);
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1192 }
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1193
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1194 static int _PyCodecRegistry_Init(void)
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	1195 {
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1196 static struct {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1197 char *name;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1198 PyMethodDef def;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1199 } methods[] =
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1200 {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1201 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1202 "strict",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1203 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1204 "strict_errors",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1205 strict_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1206 METH_O,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1207 PyDoc_STR("Implements the 'strict' error handling, which "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1208 "raises a UnicodeError on coding errors.")
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1209 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1210 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1211 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1212 "ignore",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1213 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1214 "ignore_errors",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1215 ignore_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1216 METH_O,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1217 PyDoc_STR("Implements the 'ignore' error handling, which "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1218 "ignores malformed data and continues.")
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1219 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1220 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1221 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1222 "replace",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1223 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1224 "replace_errors",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1225 replace_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1226 METH_O,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1227 PyDoc_STR("Implements the 'replace' error handling, which "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1228 "replaces malformed data with a replacement marker.")
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1229 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1230 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1231 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1232 "xmlcharrefreplace",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1233 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1234 "xmlcharrefreplace_errors",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1235 xmlcharrefreplace_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1236 METH_O,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1237 PyDoc_STR("Implements the 'xmlcharrefreplace' error handling, "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1238 "which replaces an unencodable character with the "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1239 "appropriate XML character reference.")
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1240 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1241 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1242 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1243 "backslashreplace",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1244 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1245 "backslashreplace_errors",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1246 backslashreplace_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1247 METH_O,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1248 PyDoc_STR("Implements the 'backslashreplace' error handling, "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1249 "which replaces an unencodable character with a "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1250 "backslashed escape sequence.")
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1251 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1252 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1253 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1254 "surrogatepass",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1255 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1256 "surrogatepass",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1257 surrogatepass_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1258 METH_O
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1259 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1260 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1261 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1262 "surrogateescape",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1263 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1264 "surrogateescape",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1265 surrogateescape_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1266 METH_O
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1267 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1268 }
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1269 };
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1270
31797 846b3ee7e106 Changed random calls to PyThreadState_Get() to use the macro Nicholas Bastin <nick.bastin@gmail.com> parents: 31273 diff changeset	1271 PyInterpreterState *interp = PyThreadState_GET()->interp;
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1272 PyObject *mod;
32533 201986d6d9d6 Fix a couple of signed/unsigned comparison warnings Neal Norwitz <nnorwitz@gmail.com> parents: 31797 diff changeset	1273 unsigned i;
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1274
850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1275 if (interp->codec_search_path != NULL)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1276 return 0;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1277
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1278 interp->codec_search_path = PyList_New(0);
850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1279 interp->codec_search_cache = PyDict_New();
850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1280 interp->codec_error_registry = PyDict_New();
850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1281
850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1282 if (interp->codec_error_registry) {
72510 b534b56379cb Use the new Py_ARRAY_LENGTH macro Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1283 for (i = 0; i < Py_ARRAY_LENGTH(methods); ++i) {
81031 3a86a3f1d89a Issue #15422: get rid of PyCFunction_New macro Andrew Svetlov <andrew.svetlov@gmail.com> parents: 80218 diff changeset	1284 PyObject *func = PyCFunction_NewEx(&methods[i].def, NULL, NULL);
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1285 int res;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1286 if (!func)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1287 Py_FatalError("can't initialize codec error registry");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1288 res = PyCodec_RegisterError(methods[i].name, func);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1289 Py_DECREF(func);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1290 if (res)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1291 Py_FatalError("can't initialize codec error registry");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1292 }
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1293 }
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1294
850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1295 if (interp->codec_search_path == NULL \|\|
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1296 interp->codec_search_cache == NULL \|\|
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1297 interp->codec_error_registry == NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1298 Py_FatalError("can't initialize codec registry");
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	1299
44442 fa2f7183b1af More PyImport_ImportModule -> PyImport_ImportModuleNoBlock Christian Heimes <christian@cheimes.de> parents: 44351 diff changeset	1300 mod = PyImport_ImportModuleNoBlock("encodings");
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1301 if (mod == NULL) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1302 return -1;
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1303 }
850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1304 Py_DECREF(mod);
49613 efef5fc45a9a Issue 3723: Fixed initialization of subinterpreters Christian Heimes <christian@cheimes.de> parents: 47496 diff changeset	1305 interp->codecs_initialized = 1;
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1306 return 0;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	1307 }

Mercurial > cpython-lru_cache-weakref

annotate Python/codecs.c @ 87338:d68df99d7a57