cpython-lru_cache-weakref: Python/codecs.c annotate

annotate Python/codecs.c @ 87690:66c1c9f32567 default tip

Add use_weakref option to functools.lru_cache The use_weakref option makes lru_cache create weak references to the function arguments when creating the cache key, which prevents them from staying in the memory forever even after all other references to them are gone. This is a proof of concept for http://bugs.python.org/issue19859

author	Radomir Dopieralski <devel@sheep.art.pl>
date	Tue, 03 Dec 2013 16:02:48 +0100
parents	d68df99d7a57
children

rev	line source
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	1 /* ------------------------------------------------------------------------
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	2
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	3 Python Codec Registry and support functions
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	4
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	5 Written by Marc-Andre Lemburg (mal@lemburg.com).
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	6
13828 254f8b7c4a83 Changing the CNRI copyright notice according to CNRI's instructions. Guido van Rossum <guido@python.org> parents: 13658 diff changeset	7 Copyright (c) Corporation for National Research Initiatives.
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	8
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	9 ------------------------------------------------------------------------ */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	10
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	11 #include "Python.h"
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	12 #include <ctype.h>
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	13
72918 d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16 Victor Stinner <victor.stinner@haypocalc.com> parents: 72848 diff changeset	14 const char *Py_hexdigits = "0123456789abcdef";
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16 Victor Stinner <victor.stinner@haypocalc.com> parents: 72848 diff changeset	15
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	16 /* --- Codec Registry ----------------------------------------------------- */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	17
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	18 /* Import the standard encodings package which will register the first
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	19 codec search function.
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	20
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	21 This is done in a lazy way so that the Unicode implementation does
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	22 not downgrade startup time of scripts not needing it.
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	23
12195 bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12048 diff changeset	24 ImportErrors are silently ignored by this function. Only one try is
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12048 diff changeset	25 made.
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	26
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	27 */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	28
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	29 static int _PyCodecRegistry_Init(void); /* Forward */
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	30
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	31 int PyCodec_Register(PyObject *search_function)
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	32 {
31797 846b3ee7e106 Changed random calls to PyThreadState_Get() to use the macro Nicholas Bastin <nick.bastin@gmail.com> parents: 31273 diff changeset	33 PyInterpreterState *interp = PyThreadState_GET()->interp;
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	34 if (interp->codec_search_path == NULL && _PyCodecRegistry_Init())
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	35 goto onError;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	36 if (search_function == NULL) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	37 PyErr_BadArgument();
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	38 goto onError;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	39 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	40 if (!PyCallable_Check(search_function)) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	41 PyErr_SetString(PyExc_TypeError, "argument must be callable");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	42 goto onError;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	43 }
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	44 return PyList_Append(interp->codec_search_path, search_function);
12195 bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12048 diff changeset	45
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12048 diff changeset	46 onError:
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12048 diff changeset	47 return -1;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	48 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	49
12248 70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that Guido van Rossum <guido@python.org> parents: 12195 diff changeset	50 /* Convert a string to a normalized Python string: all characters are
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that Guido van Rossum <guido@python.org> parents: 12195 diff changeset	51 converted to lower case, spaces are replaced with underscores. */
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that Guido van Rossum <guido@python.org> parents: 12195 diff changeset	52
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	53 static
12248 70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that Guido van Rossum <guido@python.org> parents: 12195 diff changeset	54 PyObject normalizestring(const char string)
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	55 {
85157 e7f6cef7a4cc Issue #18722: Remove uses of the "register" keyword in C code. Antoine Pitrou <solipsis@pitrou.net> parents: 84571 diff changeset	56 size_t i;
13096 8c8de5b736dc Trent Mick's Win64 changes: size_t vs. int or long; also some overflow Guido van Rossum <guido@python.org> parents: 12648 diff changeset	57 size_t len = strlen(string);
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	58 char *p;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	59 PyObject *v;
43603 dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee: Guido van Rossum <guido@python.org> parents: 42534 diff changeset	60
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	61 if (len > PY_SSIZE_T_MAX) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	62 PyErr_SetString(PyExc_OverflowError, "string is too large");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	63 return NULL;
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	64 }
43603 dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee: Guido van Rossum <guido@python.org> parents: 42534 diff changeset	65
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee: Guido van Rossum <guido@python.org> parents: 42534 diff changeset	66 p = PyMem_Malloc(len + 1);
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee: Guido van Rossum <guido@python.org> parents: 42534 diff changeset	67 if (p == NULL)
84571 4975bcd67aa7 Issue #18408: normalizestring() now raises MemoryError on memory allocation failure Victor Stinner <victor.stinner@gmail.com> parents: 81031 diff changeset	68 return PyErr_NoMemory();
12248 70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that Guido van Rossum <guido@python.org> parents: 12195 diff changeset	69 for (i = 0; i < len; i++) {
85157 e7f6cef7a4cc Issue #18722: Remove uses of the "register" keyword in C code. Antoine Pitrou <solipsis@pitrou.net> parents: 84571 diff changeset	70 char ch = string[i];
12248 70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that Guido van Rossum <guido@python.org> parents: 12195 diff changeset	71 if (ch == ' ')
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that Guido van Rossum <guido@python.org> parents: 12195 diff changeset	72 ch = '-';
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that Guido van Rossum <guido@python.org> parents: 12195 diff changeset	73 else
71489 92d02de91cc9 Issue #1813: Fix codec lookup under Turkish locales. Antoine Pitrou <solipsis@pitrou.net> parents: 64590 diff changeset	74 ch = Py_TOLOWER(Py_CHARMASK(ch));
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	75 p[i] = ch;
12248 70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that Guido van Rossum <guido@python.org> parents: 12195 diff changeset	76 }
43603 dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee: Guido van Rossum <guido@python.org> parents: 42534 diff changeset	77 p[i] = '\0';
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee: Guido van Rossum <guido@python.org> parents: 42534 diff changeset	78 v = PyUnicode_FromString(p);
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee: Guido van Rossum <guido@python.org> parents: 42534 diff changeset	79 if (v == NULL)
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee: Guido van Rossum <guido@python.org> parents: 42534 diff changeset	80 return NULL;
dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee: Guido van Rossum <guido@python.org> parents: 42534 diff changeset	81 PyMem_Free(p);
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	82 return v;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	83 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	84
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	85 /* Lookup the given encoding and return a tuple providing the codec
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	86 facilities.
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	87
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	88 The encoding string is looked up converted to all lower-case
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	89 characters. This makes encodings looked up through this mechanism
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	90 effectively case-insensitive.
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	91
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	92 If no codec is found, a LookupError is set and NULL returned.
12195 bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12048 diff changeset	93
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12048 diff changeset	94 As side effect, this tries to load the encodings package, if not
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12048 diff changeset	95 yet done. This is part of the lazy load strategy for the encodings
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12048 diff changeset	96 package.
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12048 diff changeset	97
bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12048 diff changeset	98 */
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	99
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	100 PyObject _PyCodec_Lookup(const char encoding)
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	101 {
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	102 PyInterpreterState *interp;
12048 f87b56eebacc Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12025 diff changeset	103 PyObject result, args = NULL, *v;
38453 29ae5e0fc348 Much-needed merge (using svnmerge.py this time) of trunk changes into p3yk. Thomas Wouters <thomas@python.org> parents: 37879 diff changeset	104 Py_ssize_t i, len;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	105
12648 2a53f5dc0590 M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12248 diff changeset	106 if (encoding == NULL) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	107 PyErr_BadArgument();
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	108 goto onError;
12648 2a53f5dc0590 M.-A. Lemburg <mal@lemburg.com>: Fred Drake <fdrake@acm.org> parents: 12248 diff changeset	109 }
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	110
31797 846b3ee7e106 Changed random calls to PyThreadState_Get() to use the macro Nicholas Bastin <nick.bastin@gmail.com> parents: 31273 diff changeset	111 interp = PyThreadState_GET()->interp;
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	112 if (interp->codec_search_path == NULL && _PyCodecRegistry_Init())
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	113 goto onError;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	114
12248 70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that Guido van Rossum <guido@python.org> parents: 12195 diff changeset	115 /* Convert the encoding to a normalized Python string: all
13620 5b727ab37092 Spelling fixes supplied by Rob W. W. Hooft. All these are fixes in either Thomas Wouters <thomas@python.org> parents: 13330 diff changeset	116 characters are converted to lower case, spaces and hyphens are
12248 70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that Guido van Rossum <guido@python.org> parents: 12195 diff changeset	117 replaced with underscores. */
70154e2e9c89 Marc-Andre's third try at this bulk patch seems to work (except that Guido van Rossum <guido@python.org> parents: 12195 diff changeset	118 v = normalizestring(encoding);
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	119 if (v == NULL)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	120 goto onError;
43603 dd03aa741f56 This is the uncontroversial half of patch 1263 by Thomas Lee: Guido van Rossum <guido@python.org> parents: 42534 diff changeset	121 PyUnicode_InternInPlace(&v);
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	122
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	123 /* First, try to lookup the name in the registry dictionary */
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	124 result = PyDict_GetItem(interp->codec_search_cache, v);
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	125 if (result != NULL) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	126 Py_INCREF(result);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	127 Py_DECREF(v);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	128 return result;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	129 }
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	130
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	131 /* Next, scan the search functions in order of registration */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	132 args = PyTuple_New(1);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	133 if (args == NULL)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	134 goto onError;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	135 PyTuple_SET_ITEM(args,0,v);
12048 f87b56eebacc Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12025 diff changeset	136
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	137 len = PyList_Size(interp->codec_search_path);
12048 f87b56eebacc Marc-Andre Lemburg: Guido van Rossum <guido@python.org> parents: 12025 diff changeset	138 if (len < 0)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	139 goto onError;
12195 bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12048 diff changeset	140 if (len == 0) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	141 PyErr_SetString(PyExc_LookupError,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	142 "no codec search functions registered: "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	143 "can't find encoding");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	144 goto onError;
12195 bc861add5d71 Marc-Andre Lemburg: Error reporting in the codec registry and lookup Guido van Rossum <guido@python.org> parents: 12048 diff changeset	145 }
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	146
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	147 for (i = 0; i < len; i++) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	148 PyObject *func;
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	149
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	150 func = PyList_GetItem(interp->codec_search_path, i);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	151 if (func == NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	152 goto onError;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	153 result = PyEval_CallObject(func, args);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	154 if (result == NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	155 goto onError;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	156 if (result == Py_None) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	157 Py_DECREF(result);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	158 continue;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	159 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	160 if (!PyTuple_Check(result) \|\| PyTuple_GET_SIZE(result) != 4) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	161 PyErr_SetString(PyExc_TypeError,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	162 "codec search functions must return 4-tuples");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	163 Py_DECREF(result);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	164 goto onError;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	165 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	166 break;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	167 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	168 if (i == len) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	169 /* XXX Perhaps we should cache misses too ? */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	170 PyErr_Format(PyExc_LookupError,
25441 d32588c88633 Add encoding name in LookupError. Fixes #615013. Will backport to 2.2. Martin v. Löwis <martin@v.loewis.de> parents: 25095 diff changeset	171 "unknown encoding: %s", encoding);
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	172 goto onError;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	173 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	174
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	175 /* Cache and return the result */
42534 5f9e79d1e158 Handle error Neal Norwitz <nnorwitz@gmail.com> parents: 42057 diff changeset	176 if (PyDict_SetItem(interp->codec_search_cache, v, result) < 0) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	177 Py_DECREF(result);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	178 goto onError;
42534 5f9e79d1e158 Handle error Neal Norwitz <nnorwitz@gmail.com> parents: 42057 diff changeset	179 }
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	180 Py_DECREF(args);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	181 return result;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	182
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	183 onError:
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	184 Py_XDECREF(args);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	185 return NULL;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	186 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	187
47496 2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	188 /* Codec registry encoding check API. */
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	189
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	190 int PyCodec_KnownEncoding(const char *encoding)
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	191 {
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	192 PyObject *codecs;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	193
47496 2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	194 codecs = _PyCodec_Lookup(encoding);
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	195 if (!codecs) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	196 PyErr_Clear();
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	197 return 0;
47496 2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	198 }
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	199 else {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	200 Py_DECREF(codecs);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	201 return 1;
47496 2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	202 }
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	203 }
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	204
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	205 static
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	206 PyObject args_tuple(PyObject object,
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	207 const char *errors)
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	208 {
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	209 PyObject *args;
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	210
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	211 args = PyTuple_New(1 + (errors != NULL));
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	212 if (args == NULL)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	213 return NULL;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	214 Py_INCREF(object);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	215 PyTuple_SET_ITEM(args,0,object);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	216 if (errors) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	217 PyObject *v;
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	218
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	219 v = PyUnicode_FromString(errors);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	220 if (v == NULL) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	221 Py_DECREF(args);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	222 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	223 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	224 PyTuple_SET_ITEM(args, 1, v);
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	225 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	226 return args;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	227 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	228
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	229 /* Helper function to get a codec item */
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	230
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	231 static
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	232 PyObject codec_getitem(const char encoding, int index)
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	233 {
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	234 PyObject *codecs;
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	235 PyObject *v;
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	236
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	237 codecs = _PyCodec_Lookup(encoding);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	238 if (codecs == NULL)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	239 return NULL;
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	240 v = PyTuple_GET_ITEM(codecs, index);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	241 Py_DECREF(codecs);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	242 Py_INCREF(v);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	243 return v;
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	244 }
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	245
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	246 /* Helper function to create an incremental codec. */
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	247
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	248 static
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	249 PyObject codec_getincrementalcodec(const char encoding,
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	250 const char *errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	251 const char *attrname)
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	252 {
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	253 PyObject codecs, ret, *inccodec;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	254
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	255 codecs = _PyCodec_Lookup(encoding);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	256 if (codecs == NULL)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	257 return NULL;
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	258 inccodec = PyObject_GetAttrString(codecs, attrname);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	259 Py_DECREF(codecs);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	260 if (inccodec == NULL)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	261 return NULL;
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	262 if (errors)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	263 ret = PyObject_CallFunction(inccodec, "s", errors);
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	264 else
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	265 ret = PyObject_CallFunction(inccodec, NULL);
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	266 Py_DECREF(inccodec);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	267 return ret;
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	268 }
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	269
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	270 /* Helper function to create a stream codec. */
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	271
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	272 static
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	273 PyObject codec_getstreamcodec(const char encoding,
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	274 PyObject *stream,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	275 const char *errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	276 const int index)
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	277 {
39442 654c380cf8b9 Merged revisions 46753-51188 via svnmerge from Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	278 PyObject codecs, streamcodec, *codeccls;
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	279
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	280 codecs = _PyCodec_Lookup(encoding);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	281 if (codecs == NULL)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	282 return NULL;
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	283
39442 654c380cf8b9 Merged revisions 46753-51188 via svnmerge from Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	284 codeccls = PyTuple_GET_ITEM(codecs, index);
654c380cf8b9 Merged revisions 46753-51188 via svnmerge from Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	285 if (errors != NULL)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	286 streamcodec = PyObject_CallFunction(codeccls, "Os", stream, errors);
39442 654c380cf8b9 Merged revisions 46753-51188 via svnmerge from Thomas Wouters <thomas@python.org> parents: 38453 diff changeset	287 else
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	288 streamcodec = PyObject_CallFunction(codeccls, "O", stream);
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	289 Py_DECREF(codecs);
f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	290 return streamcodec;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	291 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	292
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	293 /* Convenience APIs to query the Codec registry.
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	294
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	295 All APIs return a codec object with incremented refcount.
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	296
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	297 */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	298
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	299 PyObject PyCodec_Encoder(const char encoding)
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	300 {
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	301 return codec_getitem(encoding, 0);
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	302 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	303
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	304 PyObject PyCodec_Decoder(const char encoding)
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	305 {
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	306 return codec_getitem(encoding, 1);
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	307 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	308
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36751 diff changeset	309 PyObject PyCodec_IncrementalEncoder(const char encoding,
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	310 const char *errors)
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36751 diff changeset	311 {
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	312 return codec_getincrementalcodec(encoding, errors, "incrementalencoder");
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36751 diff changeset	313 }
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36751 diff changeset	314
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36751 diff changeset	315 PyObject PyCodec_IncrementalDecoder(const char encoding,
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	316 const char *errors)
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36751 diff changeset	317 {
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	318 return codec_getincrementalcodec(encoding, errors, "incrementaldecoder");
36996 7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36751 diff changeset	319 }
7fa4c29d7030 Patch #1436130: codecs.lookup() now returns a CodecInfo object (a subclass Walter Dörwald <walter@livinglogic.de> parents: 36751 diff changeset	320
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	321 PyObject PyCodec_StreamReader(const char encoding,
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	322 PyObject *stream,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	323 const char *errors)
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	324 {
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	325 return codec_getstreamcodec(encoding, stream, errors, 2);
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	326 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	327
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	328 PyObject PyCodec_StreamWriter(const char encoding,
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	329 PyObject *stream,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	330 const char *errors)
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	331 {
37879 f731859e9b3b Merge p3yk branch with the trunk up to revision 45595. This breaks a fair Thomas Wouters <thomas@python.org> parents: 36996 diff changeset	332 return codec_getstreamcodec(encoding, stream, errors, 3);
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	333 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	334
87084 854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	335 /* Helper that tries to ensure the reported exception chain indicates the
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	336 * codec that was invoked to trigger the failure without changing the type
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	337 * of the exception raised.
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	338 */
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	339 static void
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	340 wrap_codec_error(const char *operation,
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	341 const char *encoding)
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	342 {
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	343 /* TrySetFromCause will replace the active exception with a suitably
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	344 * updated clone if it can, otherwise it will leave the original
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	345 * exception alone.
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	346 */
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	347 _PyErr_TrySetFromCause("%s with '%s' codec failed",
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	348 operation, encoding);
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	349 }
854a2cea31b9 Close #17828: better handling of codec errors Nick Coghlan <ncoghlan@gmail.com> parents: 86484 diff changeset	350
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	351 /* Encode an object (e.g. an Unicode object) using the given encoding
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	352 and return the resulting encoded object (usually a Python string).
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	353
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	354 errors is passed to the encoder factory as argument if non-NULL. */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	355
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	356 static PyObject *
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	357 _PyCodec_EncodeInternal(PyObject *object,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	358 PyObject *encoder,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	359 const char *encoding,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	360 const char *errors)
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	361 {
35939 79374eb6db51 Fix a few more ref leaks. Backport candidate Neal Norwitz <nnorwitz@gmail.com> parents: 32533 diff changeset	362 PyObject args = NULL, result = NULL;
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	363 PyObject *v = NULL;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	364
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	365 args = args_tuple(object, errors);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	366 if (args == NULL)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	367 goto onError;
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	368
0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	369 result = PyEval_CallObject(encoder, args);
87109 4ea622c085ca Close 19609: narrow scope of codec exc chaining Nick Coghlan <ncoghlan@gmail.com> parents: 87084 diff changeset	370 if (result == NULL) {
4ea622c085ca Close 19609: narrow scope of codec exc chaining Nick Coghlan <ncoghlan@gmail.com> parents: 87084 diff changeset	371 wrap_codec_error("encoding", encoding);
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	372 goto onError;
87109 4ea622c085ca Close 19609: narrow scope of codec exc chaining Nick Coghlan <ncoghlan@gmail.com> parents: 87084 diff changeset	373 }
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	374
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	375 if (!PyTuple_Check(result) \|\|
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	376 PyTuple_GET_SIZE(result) != 2) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	377 PyErr_SetString(PyExc_TypeError,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	378 "encoder must return a tuple (object, integer)");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	379 goto onError;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	380 }
47496 2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	381 v = PyTuple_GET_ITEM(result,0);
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	382 Py_INCREF(v);
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	383 /* We don't check or use the second (integer) entry. */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	384
47496 2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	385 Py_DECREF(args);
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	386 Py_DECREF(encoder);
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	387 Py_DECREF(result);
2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	388 return v;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	389
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	390 onError:
35939 79374eb6db51 Fix a few more ref leaks. Backport candidate Neal Norwitz <nnorwitz@gmail.com> parents: 32533 diff changeset	391 Py_XDECREF(result);
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	392 Py_XDECREF(args);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	393 Py_XDECREF(encoder);
47496 2284fa89ab08 Move the codec decode type checks to bytes/bytearray.decode(). Marc-André Lemburg <mal@egenix.com> parents: 47301 diff changeset	394 return NULL;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	395 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	396
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	397 /* Decode an object (usually a Python string) using the given encoding
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	398 and return an equivalent object (e.g. an Unicode object).
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	399
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	400 errors is passed to the decoder factory as argument if non-NULL. */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	401
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	402 static PyObject *
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	403 _PyCodec_DecodeInternal(PyObject *object,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	404 PyObject *decoder,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	405 const char *encoding,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	406 const char *errors)
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	407 {
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	408 PyObject args = NULL, result = NULL;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	409 PyObject *v;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	410
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	411 args = args_tuple(object, errors);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	412 if (args == NULL)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	413 goto onError;
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	414
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	415 result = PyEval_CallObject(decoder,args);
87109 4ea622c085ca Close 19609: narrow scope of codec exc chaining Nick Coghlan <ncoghlan@gmail.com> parents: 87084 diff changeset	416 if (result == NULL) {
4ea622c085ca Close 19609: narrow scope of codec exc chaining Nick Coghlan <ncoghlan@gmail.com> parents: 87084 diff changeset	417 wrap_codec_error("decoding", encoding);
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	418 goto onError;
87109 4ea622c085ca Close 19609: narrow scope of codec exc chaining Nick Coghlan <ncoghlan@gmail.com> parents: 87084 diff changeset	419 }
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	420 if (!PyTuple_Check(result) \|\|
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	421 PyTuple_GET_SIZE(result) != 2) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	422 PyErr_SetString(PyExc_TypeError,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	423 "decoder must return a tuple (object,integer)");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	424 goto onError;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	425 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	426 v = PyTuple_GET_ITEM(result,0);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	427 Py_INCREF(v);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	428 /* We don't check or use the second (integer) entry. */
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	429
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	430 Py_DECREF(args);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	431 Py_DECREF(decoder);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	432 Py_DECREF(result);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	433 return v;
43777 0d462d789b18 Merging the py3k-pep3137 branch back into the py3k branch. Guido van Rossum <guido@python.org> parents: 43603 diff changeset	434
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	435 onError:
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	436 Py_XDECREF(args);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	437 Py_XDECREF(decoder);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	438 Py_XDECREF(result);
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	439 return NULL;
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	440 }
d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	441
87338 d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	442 /* Generic encoding/decoding API */
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	443 PyObject PyCodec_Encode(PyObject object,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	444 const char *encoding,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	445 const char *errors)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	446 {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	447 PyObject *encoder;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	448
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	449 encoder = PyCodec_Encoder(encoding);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	450 if (encoder == NULL)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	451 return NULL;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	452
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	453 return _PyCodec_EncodeInternal(object, encoder, encoding, errors);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	454 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	455
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	456 PyObject PyCodec_Decode(PyObject object,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	457 const char *encoding,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	458 const char *errors)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	459 {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	460 PyObject *decoder;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	461
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	462 decoder = PyCodec_Decoder(encoding);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	463 if (decoder == NULL)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	464 return NULL;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	465
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	466 return _PyCodec_DecodeInternal(object, decoder, encoding, errors);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	467 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	468
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	469 /* Text encoding/decoding API */
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	470 static
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	471 PyObject codec_getitem_checked(const char encoding,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	472 const char *operation_name,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	473 int index)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	474 {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	475 _Py_IDENTIFIER(_is_text_encoding);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	476 PyObject *codec;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	477 PyObject *attr;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	478 PyObject *v;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	479 int is_text_codec;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	480
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	481 codec = _PyCodec_Lookup(encoding);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	482 if (codec == NULL)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	483 return NULL;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	484
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	485 /* Backwards compatibility: assume any raw tuple describes a text
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	486 * encoding, and the same for anything lacking the private
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	487 * attribute.
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	488 */
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	489 if (!PyTuple_CheckExact(codec)) {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	490 attr = _PyObject_GetAttrId(codec, &PyId__is_text_encoding);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	491 if (attr == NULL) {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	492 if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	493 PyErr_Clear();
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	494 } else {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	495 Py_DECREF(codec);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	496 return NULL;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	497 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	498 } else {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	499 is_text_codec = PyObject_IsTrue(attr);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	500 Py_DECREF(attr);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	501 if (!is_text_codec) {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	502 Py_DECREF(codec);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	503 PyErr_Format(PyExc_LookupError,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	504 "'%.400s' is not a text encoding; "
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	505 "use codecs.%s() to handle arbitrary codecs",
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	506 encoding, operation_name);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	507 return NULL;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	508 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	509 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	510 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	511
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	512 v = PyTuple_GET_ITEM(codec, index);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	513 Py_DECREF(codec);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	514 Py_INCREF(v);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	515 return v;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	516 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	517
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	518 static PyObject * _PyCodec_TextEncoder(const char *encoding)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	519 {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	520 return codec_getitem_checked(encoding, "encode", 0);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	521 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	522
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	523 static PyObject * _PyCodec_TextDecoder(const char *encoding)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	524 {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	525 return codec_getitem_checked(encoding, "decode", 1);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	526 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	527
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	528 PyObject _PyCodec_EncodeText(PyObject object,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	529 const char *encoding,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	530 const char *errors)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	531 {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	532 PyObject *encoder;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	533
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	534 encoder = _PyCodec_TextEncoder(encoding);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	535 if (encoder == NULL)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	536 return NULL;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	537
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	538 return _PyCodec_EncodeInternal(object, encoder, encoding, errors);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	539 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	540
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	541 PyObject _PyCodec_DecodeText(PyObject object,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	542 const char *encoding,
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	543 const char *errors)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	544 {
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	545 PyObject *decoder;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	546
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	547 decoder = _PyCodec_TextDecoder(encoding);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	548 if (decoder == NULL)
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	549 return NULL;
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	550
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	551 return _PyCodec_DecodeInternal(object, decoder, encoding, errors);
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	552 }
d68df99d7a57 Issue #19619: Blacklist non-text codecs in method API Nick Coghlan <ncoghlan@gmail.com> parents: 87261 diff changeset	553
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	554 /* Register the error handling callback function error under the name
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	555 name. This function will be called by the codec when it encounters
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	556 an unencodable characters/undecodable bytes and doesn't know the
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	557 callback name, when name is specified as the error parameter
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	558 in the call to the encode/decode function.
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	559 Return 0 on success, -1 on error */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	560 int PyCodec_RegisterError(const char name, PyObject error)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	561 {
31797 846b3ee7e106 Changed random calls to PyThreadState_Get() to use the macro Nicholas Bastin <nick.bastin@gmail.com> parents: 31273 diff changeset	562 PyInterpreterState *interp = PyThreadState_GET()->interp;
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	563 if (interp->codec_search_path == NULL && _PyCodecRegistry_Init())
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	564 return -1;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	565 if (!PyCallable_Check(error)) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	566 PyErr_SetString(PyExc_TypeError, "handler must be callable");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	567 return -1;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	568 }
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	569 return PyDict_SetItemString(interp->codec_error_registry,
86484 3055e61f1e66 Issue #1772673: The type of `char` arguments now changed to `const char`. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	570 name, error);
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	571 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	572
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	573 /* Lookup the error handling callback function registered under the
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	574 name error. As a special case NULL can be passed, in which case
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	575 the error handling callback for strict encoding will be returned. */
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	576 PyObject PyCodec_LookupError(const char name)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	577 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	578 PyObject *handler = NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	579
31797 846b3ee7e106 Changed random calls to PyThreadState_Get() to use the macro Nicholas Bastin <nick.bastin@gmail.com> parents: 31273 diff changeset	580 PyInterpreterState *interp = PyThreadState_GET()->interp;
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	581 if (interp->codec_search_path == NULL && _PyCodecRegistry_Init())
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	582 return NULL;
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	583
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	584 if (name==NULL)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	585 name = "strict";
86484 3055e61f1e66 Issue #1772673: The type of `char` arguments now changed to `const char`. Serhiy Storchaka <storchaka@gmail.com> parents: 85157 diff changeset	586 handler = PyDict_GetItemString(interp->codec_error_registry, name);
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	587 if (!handler)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	588 PyErr_Format(PyExc_LookupError, "unknown error handler name '%.400s'", name);
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	589 else
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	590 Py_INCREF(handler);
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	591 return handler;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	592 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	593
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	594 static void wrong_exception_type(PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	595 {
72925 7109f31300fb Rename _Py_identifier to _Py_IDENTIFIER. Martin v. Löwis <martin@v.loewis.de> parents: 72918 diff changeset	596 _Py_IDENTIFIER(__class__);
7109f31300fb Rename _Py_identifier to _Py_IDENTIFIER. Martin v. Löwis <martin@v.loewis.de> parents: 72918 diff changeset	597 _Py_IDENTIFIER(__name__);
72848 81380082d216 Use identifier API for PyObject_GetAttrString. Martin v. Löwis <martin@v.loewis.de> parents: 72591 diff changeset	598 PyObject *type = _PyObject_GetAttrId(exc, &PyId___class__);
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	599 if (type != NULL) {
72848 81380082d216 Use identifier API for PyObject_GetAttrString. Martin v. Löwis <martin@v.loewis.de> parents: 72591 diff changeset	600 PyObject *name = _PyObject_GetAttrId(type, &PyId___name__);
41878 089f449a20cd Change PyErr_Format() to generate a unicode string (by using Walter Dörwald <walter@livinglogic.de> parents: 41587 diff changeset	601 Py_DECREF(type);
089f449a20cd Change PyErr_Format() to generate a unicode string (by using Walter Dörwald <walter@livinglogic.de> parents: 41587 diff changeset	602 if (name != NULL) {
089f449a20cd Change PyErr_Format() to generate a unicode string (by using Walter Dörwald <walter@livinglogic.de> parents: 41587 diff changeset	603 PyErr_Format(PyExc_TypeError,
089f449a20cd Change PyErr_Format() to generate a unicode string (by using Walter Dörwald <walter@livinglogic.de> parents: 41587 diff changeset	604 "don't know how to handle %S in error callback", name);
089f449a20cd Change PyErr_Format() to generate a unicode string (by using Walter Dörwald <walter@livinglogic.de> parents: 41587 diff changeset	605 Py_DECREF(name);
089f449a20cd Change PyErr_Format() to generate a unicode string (by using Walter Dörwald <walter@livinglogic.de> parents: 41587 diff changeset	606 }
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	607 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	608 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	609
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	610 PyObject PyCodec_StrictErrors(PyObject exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	611 {
36751 8e04de2480e4 PEP 352 implementation. Creates a new base class, BaseException, which has an Brett Cannon <bcannon@gmail.com> parents: 36712 diff changeset	612 if (PyExceptionInstance_Check(exc))
8e04de2480e4 PEP 352 implementation. Creates a new base class, BaseException, which has an Brett Cannon <bcannon@gmail.com> parents: 36712 diff changeset	613 PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	614 else
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	615 PyErr_SetString(PyExc_TypeError, "codec must pass exception instance");
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	616 return NULL;
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	617 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	618
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	619
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	620 PyObject PyCodec_IgnoreErrors(PyObject exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	621 {
36501 7fdc639bc5b4 Merge ssize_t branch. Martin v. Löwis <martin@v.loewis.de> parents: 35939 diff changeset	622 Py_ssize_t end;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	623 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	624 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	625 return NULL;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	626 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	627 else if (PyObject_IsInstance(exc, PyExc_UnicodeDecodeError)) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	628 if (PyUnicodeDecodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	629 return NULL;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	630 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	631 else if (PyObject_IsInstance(exc, PyExc_UnicodeTranslateError)) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	632 if (PyUnicodeTranslateError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	633 return NULL;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	634 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	635 else {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	636 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	637 return NULL;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	638 }
73800 6407294eb3c6 PyCodec_IgnoreErrors() avoids the deprecated "u#" format Victor Stinner <victor.stinner@haypocalc.com> parents: 73359 diff changeset	639 return Py_BuildValue("(Nn)", PyUnicode_New(0, 0), end);
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	640 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	641
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	642
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	643 PyObject PyCodec_ReplaceErrors(PyObject exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	644 {
72475 8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	645 Py_ssize_t start, end, i, len;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	646
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	647 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	648 PyObject *res;
72475 8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	649 int kind;
8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	650 void *data;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	651 if (PyUnicodeEncodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	652 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	653 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	654 return NULL;
72475 8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	655 len = end - start;
8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	656 res = PyUnicode_New(len, '?');
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	657 if (res == NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	658 return NULL;
72475 8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	659 kind = PyUnicode_KIND(res);
8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	660 data = PyUnicode_DATA(res);
8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	661 for (i = 0; i < len; ++i)
8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	662 PyUnicode_WRITE(kind, data, i, '?');
76573 1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1) Victor Stinner <victor.stinner@gmail.com> parents: 74512 diff changeset	663 assert(_PyUnicode_CheckConsistency(res, 1));
72475 8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	664 return Py_BuildValue("(Nn)", res, end);
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	665 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	666 else if (PyObject_IsInstance(exc, PyExc_UnicodeDecodeError)) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	667 if (PyUnicodeDecodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	668 return NULL;
72591 611c57aa694a PyCodec_ReplaceErrors() uses "C" format instead of "u#" to build result Victor Stinner <victor.stinner@haypocalc.com> parents: 72510 diff changeset	669 return Py_BuildValue("(Cn)",
611c57aa694a PyCodec_ReplaceErrors() uses "C" format instead of "u#" to build result Victor Stinner <victor.stinner@haypocalc.com> parents: 72510 diff changeset	670 (int)Py_UNICODE_REPLACEMENT_CHARACTER,
611c57aa694a PyCodec_ReplaceErrors() uses "C" format instead of "u#" to build result Victor Stinner <victor.stinner@haypocalc.com> parents: 72510 diff changeset	671 end);
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	672 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	673 else if (PyObject_IsInstance(exc, PyExc_UnicodeTranslateError)) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	674 PyObject *res;
72475 8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	675 int kind;
8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	676 void *data;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	677 if (PyUnicodeTranslateError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	678 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	679 if (PyUnicodeTranslateError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	680 return NULL;
72475 8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	681 len = end - start;
8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	682 res = PyUnicode_New(len, Py_UNICODE_REPLACEMENT_CHARACTER);
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	683 if (res == NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	684 return NULL;
72475 8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	685 kind = PyUnicode_KIND(res);
8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	686 data = PyUnicode_DATA(res);
8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	687 for (i=0; i < len; i++)
8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	688 PyUnicode_WRITE(kind, data, i, Py_UNICODE_REPLACEMENT_CHARACTER);
76573 1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1) Victor Stinner <victor.stinner@gmail.com> parents: 74512 diff changeset	689 assert(_PyUnicode_CheckConsistency(res, 1));
72475 8beaa9a37387 Implement PEP 393. Martin v. Löwis <martin@v.loewis.de> parents: 71489 diff changeset	690 return Py_BuildValue("(Nn)", res, end);
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	691 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	692 else {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	693 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	694 return NULL;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	695 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	696 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	697
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	698 PyObject PyCodec_XMLCharRefReplaceErrors(PyObject exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	699 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	700 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	701 PyObject *restuple;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	702 PyObject *object;
73358 2df7d749ee3a PyCodec_XMLCharRefReplaceError(): Remove unused variable Victor Stinner <victor.stinner@haypocalc.com> parents: 73351 diff changeset	703 Py_ssize_t i;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	704 Py_ssize_t start;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	705 Py_ssize_t end;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	706 PyObject *res;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	707 unsigned char *outp;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	708 int ressize;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	709 Py_UCS4 ch;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	710 if (PyUnicodeEncodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	711 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	712 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	713 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	714 if (!(object = PyUnicodeEncodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	715 return NULL;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	716 for (i = start, ressize = 0; i < end; ++i) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	717 /* object is guaranteed to be "ready" */
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	718 ch = PyUnicode_READ_CHAR(object, i);
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	719 if (ch<10)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	720 ressize += 2+1+1;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	721 else if (ch<100)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	722 ressize += 2+2+1;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	723 else if (ch<1000)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	724 ressize += 2+3+1;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	725 else if (ch<10000)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	726 ressize += 2+4+1;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	727 else if (ch<100000)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	728 ressize += 2+5+1;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	729 else if (ch<1000000)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	730 ressize += 2+6+1;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	731 else
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	732 ressize += 2+7+1;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	733 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	734 /* allocate replacement */
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	735 res = PyUnicode_New(ressize, 127);
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	736 if (res == NULL) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	737 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	738 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	739 }
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	740 outp = PyUnicode_1BYTE_DATA(res);
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	741 /* generate replacement */
73358 2df7d749ee3a PyCodec_XMLCharRefReplaceError(): Remove unused variable Victor Stinner <victor.stinner@haypocalc.com> parents: 73351 diff changeset	742 for (i = start; i < end; ++i) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	743 int digits;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	744 int base;
73351 2bec7c452b39 Fix C89 incompatibility. Martin v. Löwis <martin@v.loewis.de> parents: 73350 diff changeset	745 ch = PyUnicode_READ_CHAR(object, i);
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	746 *outp++ = '&';
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	747 *outp++ = '#';
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	748 if (ch<10) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	749 digits = 1;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	750 base = 1;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	751 }
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	752 else if (ch<100) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	753 digits = 2;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	754 base = 10;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	755 }
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	756 else if (ch<1000) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	757 digits = 3;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	758 base = 100;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	759 }
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	760 else if (ch<10000) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	761 digits = 4;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	762 base = 1000;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	763 }
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	764 else if (ch<100000) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	765 digits = 5;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	766 base = 10000;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	767 }
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	768 else if (ch<1000000) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	769 digits = 6;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	770 base = 100000;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	771 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	772 else {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	773 digits = 7;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	774 base = 1000000;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	775 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	776 while (digits-->0) {
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	777 *outp++ = '0' + ch/base;
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	778 ch %= base;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	779 base /= 10;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	780 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	781 *outp++ = ';';
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	782 }
76573 1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1) Victor Stinner <victor.stinner@gmail.com> parents: 74512 diff changeset	783 assert(_PyUnicode_CheckConsistency(res, 1));
1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1) Victor Stinner <victor.stinner@gmail.com> parents: 74512 diff changeset	784 restuple = Py_BuildValue("(Nn)", res, end);
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	785 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	786 return restuple;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	787 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	788 else {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	789 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	790 return NULL;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	791 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	792 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	793
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	794 PyObject PyCodec_BackslashReplaceErrors(PyObject exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	795 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	796 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	797 PyObject *restuple;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	798 PyObject *object;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	799 Py_ssize_t i;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	800 Py_ssize_t start;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	801 Py_ssize_t end;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	802 PyObject *res;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	803 unsigned char *outp;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	804 int ressize;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	805 Py_UCS4 c;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	806 if (PyUnicodeEncodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	807 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	808 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	809 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	810 if (!(object = PyUnicodeEncodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	811 return NULL;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	812 for (i = start, ressize = 0; i < end; ++i) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	813 /* object is guaranteed to be "ready" */
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	814 c = PyUnicode_READ_CHAR(object, i);
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	815 if (c >= 0x10000) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	816 ressize += 1+1+8;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	817 }
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	818 else if (c >= 0x100) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	819 ressize += 1+1+4;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	820 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	821 else
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	822 ressize += 1+1+2;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	823 }
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	824 res = PyUnicode_New(ressize, 127);
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	825 if (res==NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	826 return NULL;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	827 for (i = start, outp = PyUnicode_1BYTE_DATA(res);
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	828 i < end; ++i) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	829 c = PyUnicode_READ_CHAR(object, i);
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	830 *outp++ = '\\';
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	831 if (c >= 0x00010000) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	832 *outp++ = 'U';
72918 d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16 Victor Stinner <victor.stinner@haypocalc.com> parents: 72848 diff changeset	833 *outp++ = Py_hexdigits[(c>>28)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16 Victor Stinner <victor.stinner@haypocalc.com> parents: 72848 diff changeset	834 *outp++ = Py_hexdigits[(c>>24)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16 Victor Stinner <victor.stinner@haypocalc.com> parents: 72848 diff changeset	835 *outp++ = Py_hexdigits[(c>>20)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16 Victor Stinner <victor.stinner@haypocalc.com> parents: 72848 diff changeset	836 *outp++ = Py_hexdigits[(c>>16)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16 Victor Stinner <victor.stinner@haypocalc.com> parents: 72848 diff changeset	837 *outp++ = Py_hexdigits[(c>>12)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16 Victor Stinner <victor.stinner@haypocalc.com> parents: 72848 diff changeset	838 *outp++ = Py_hexdigits[(c>>8)&0xf];
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	839 }
64590 fc76e8c617dc Issue #9804: ascii() now always represents unicode surrogate pairs as Antoine Pitrou <solipsis@pitrou.net> parents: 61466 diff changeset	840 else if (c >= 0x100) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	841 *outp++ = 'u';
72918 d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16 Victor Stinner <victor.stinner@haypocalc.com> parents: 72848 diff changeset	842 *outp++ = Py_hexdigits[(c>>12)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16 Victor Stinner <victor.stinner@haypocalc.com> parents: 72848 diff changeset	843 *outp++ = Py_hexdigits[(c>>8)&0xf];
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	844 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	845 else
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	846 *outp++ = 'x';
72918 d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16 Victor Stinner <victor.stinner@haypocalc.com> parents: 72848 diff changeset	847 *outp++ = Py_hexdigits[(c>>4)&0xf];
d76338eacf7c Issue #13088: Add shared Py_hexdigits constant to format a number into base 16 Victor Stinner <victor.stinner@haypocalc.com> parents: 72848 diff changeset	848 *outp++ = Py_hexdigits[c&0xf];
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	849 }
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	850
76573 1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1) Victor Stinner <victor.stinner@gmail.com> parents: 74512 diff changeset	851 assert(_PyUnicode_CheckConsistency(res, 1));
1439e2d1f490 Check newly created consistency using _PyUnicode_CheckConsistency(str, 1) Victor Stinner <victor.stinner@gmail.com> parents: 74512 diff changeset	852 restuple = Py_BuildValue("(Nn)", res, end);
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	853 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	854 return restuple;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	855 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	856 else {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	857 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	858 return NULL;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	859 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	860 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	861
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	862 #define ENC_UTF8 0
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	863 #define ENC_UTF16BE 1
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	864 #define ENC_UTF16LE 2
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	865 #define ENC_UTF32BE 3
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	866 #define ENC_UTF32LE 4
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	867
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	868 static int
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	869 get_standard_encoding(const char encoding, int bytelength)
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	870 {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	871 if (Py_TOLOWER(encoding[0]) == 'u' &&
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	872 Py_TOLOWER(encoding[1]) == 't' &&
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	873 Py_TOLOWER(encoding[2]) == 'f') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	874 encoding += 3;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	875 if (encoding == '-' \|\| encoding == '_' )
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	876 encoding++;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	877 if (encoding[0] == '1' && encoding[1] == '6') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	878 encoding += 2;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	879 *bytelength = 2;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	880 if (*encoding == '\0') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	881 #ifdef WORDS_BIGENDIAN
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	882 return ENC_UTF16BE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	883 #else
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	884 return ENC_UTF16LE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	885 #endif
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	886 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	887 if (encoding == '-' \|\| encoding == '_' )
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	888 encoding++;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	889 if (Py_TOLOWER(encoding[1]) == 'e' && encoding[2] == '\0') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	890 if (Py_TOLOWER(encoding[0]) == 'b')
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	891 return ENC_UTF16BE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	892 if (Py_TOLOWER(encoding[0]) == 'l')
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	893 return ENC_UTF16LE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	894 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	895 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	896 else if (encoding[0] == '3' && encoding[1] == '2') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	897 encoding += 2;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	898 *bytelength = 4;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	899 if (*encoding == '\0') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	900 #ifdef WORDS_BIGENDIAN
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	901 return ENC_UTF32BE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	902 #else
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	903 return ENC_UTF32LE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	904 #endif
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	905 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	906 if (encoding == '-' \|\| encoding == '_' )
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	907 encoding++;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	908 if (Py_TOLOWER(encoding[1]) == 'e' && encoding[2] == '\0') {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	909 if (Py_TOLOWER(encoding[0]) == 'b')
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	910 return ENC_UTF32BE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	911 if (Py_TOLOWER(encoding[0]) == 'l')
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	912 return ENC_UTF32LE;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	913 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	914 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	915 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	916 /* utf-8 */
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	917 *bytelength = 3;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	918 return ENC_UTF8;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	919 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	920
53687 52a89d723b6b Make PyCodec_SurrogateErrors static. Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	921 /* This handler is declared static until someone demonstrates
52a89d723b6b Make PyCodec_SurrogateErrors static. Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	922 a need to call it directly. */
52a89d723b6b Make PyCodec_SurrogateErrors static. Martin v. Löwis <martin@v.loewis.de> parents: 53681 diff changeset	923 static PyObject *
53965 fbcdae3e0ca1 Rename the surrogates error handler to surrogatepass. Martin v. Löwis <martin@v.loewis.de> parents: 53767 diff changeset	924 PyCodec_SurrogatePassErrors(PyObject *exc)
53681 52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	925 {
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	926 PyObject *restuple;
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	927 PyObject *object;
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	928 PyObject *encode;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	929 char *encoding;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	930 int code;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	931 int bytelength;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	932 Py_ssize_t i;
53681 52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	933 Py_ssize_t start;
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	934 Py_ssize_t end;
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	935 PyObject *res;
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	936 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	937 unsigned char *outp;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	938 if (PyUnicodeEncodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	939 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	940 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	941 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	942 if (!(object = PyUnicodeEncodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	943 return NULL;
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	944 if (!(encode = PyUnicodeEncodeError_GetEncoding(exc))) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	945 Py_DECREF(object);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	946 return NULL;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	947 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	948 if (!(encoding = PyUnicode_AsUTF8(encode))) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	949 Py_DECREF(object);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	950 Py_DECREF(encode);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	951 return NULL;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	952 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	953 code = get_standard_encoding(encoding, &bytelength);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	954 Py_DECREF(encode);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	955
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	956 res = PyBytes_FromStringAndSize(NULL, bytelength*(end-start));
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	957 if (!res) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	958 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	959 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	960 }
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	961 outp = (unsigned char*)PyBytes_AsString(res);
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	962 for (i = start; i < end; i++) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	963 /* object is guaranteed to be "ready" */
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	964 Py_UCS4 ch = PyUnicode_READ_CHAR(object, i);
80057 13dd8199c112 Issue #16330: Use surrogate-related macros Victor Stinner <victor.stinner@gmail.com> parents: 79954 diff changeset	965 if (!Py_UNICODE_IS_SURROGATE(ch)) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	966 /* Not a surrogate, fail with original exception */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	967 PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	968 Py_DECREF(res);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	969 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	970 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	971 }
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	972 switch (code) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	973 case ENC_UTF8:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	974 *outp++ = (unsigned char)(0xe0 \| (ch >> 12));
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	975 *outp++ = (unsigned char)(0x80 \| ((ch >> 6) & 0x3f));
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	976 *outp++ = (unsigned char)(0x80 \| (ch & 0x3f));
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	977 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	978 case ENC_UTF16LE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	979 *outp++ = (unsigned char) ch;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	980 *outp++ = (unsigned char)(ch >> 8);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	981 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	982 case ENC_UTF16BE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	983 *outp++ = (unsigned char)(ch >> 8);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	984 *outp++ = (unsigned char) ch;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	985 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	986 case ENC_UTF32LE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	987 *outp++ = (unsigned char) ch;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	988 *outp++ = (unsigned char)(ch >> 8);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	989 *outp++ = (unsigned char)(ch >> 16);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	990 *outp++ = (unsigned char)(ch >> 24);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	991 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	992 case ENC_UTF32BE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	993 *outp++ = (unsigned char)(ch >> 24);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	994 *outp++ = (unsigned char)(ch >> 16);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	995 *outp++ = (unsigned char)(ch >> 8);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	996 *outp++ = (unsigned char) ch;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	997 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	998 }
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	999 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1000 restuple = Py_BuildValue("(On)", res, end);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1001 Py_DECREF(res);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1002 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1003 return restuple;
53681 52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	1004 }
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	1005 else if (PyObject_IsInstance(exc, PyExc_UnicodeDecodeError)) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1006 unsigned char *p;
73359 e3b6458056c7 Avoid the Py_UNICODE type in codecs.c Victor Stinner <victor.stinner@haypocalc.com> parents: 73358 diff changeset	1007 Py_UCS4 ch = 0;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1008 if (PyUnicodeDecodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1009 return NULL;
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1010 if (PyUnicodeDecodeError_GetEnd(exc, &end))
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1011 return NULL;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1012 if (!(object = PyUnicodeDecodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1013 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1014 if (!(p = (unsigned char*)PyBytes_AsString(object))) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1015 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1016 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1017 }
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1018 if (!(encode = PyUnicodeDecodeError_GetEncoding(exc))) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1019 Py_DECREF(object);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1020 return NULL;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1021 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1022 if (!(encoding = PyUnicode_AsUTF8(encode))) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1023 Py_DECREF(object);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1024 Py_DECREF(encode);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1025 return NULL;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1026 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1027 code = get_standard_encoding(encoding, &bytelength);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1028 Py_DECREF(encode);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1029
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1030 /* Try decoding a single surrogate character. If
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1031 there are more, let the codec call us again. */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1032 p += start;
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1033 if (PyBytes_GET_SIZE(object) - start >= bytelength) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1034 switch (code) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1035 case ENC_UTF8:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1036 if ((p[0] & 0xf0) == 0xe0 &&
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1037 (p[1] & 0xc0) == 0x80 &&
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1038 (p[2] & 0xc0) == 0x80) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1039 /* it's a three-byte code */
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1040 ch = ((p[0] & 0x0f) << 12) + ((p[1] & 0x3f) << 6) + (p[2] & 0x3f);
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1041 }
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1042 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1043 case ENC_UTF16LE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1044 ch = p[1] << 8 \| p[0];
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1045 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1046 case ENC_UTF16BE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1047 ch = p[0] << 8 \| p[1];
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1048 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1049 case ENC_UTF32LE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1050 ch = (p[3] << 24) \| (p[2] << 16) \| (p[1] << 8) \| p[0];
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1051 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1052 case ENC_UTF32BE:
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1053 ch = (p[0] << 24) \| (p[1] << 16) \| (p[2] << 8) \| p[3];
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1054 break;
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1055 }
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1056 }
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1057
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1058 Py_DECREF(object);
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1059 if (!Py_UNICODE_IS_SURROGATE(ch)) {
0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1060 /* it's not a surrogate - fail */
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1061 PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1062 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1063 }
73359 e3b6458056c7 Avoid the Py_UNICODE type in codecs.c Victor Stinner <victor.stinner@haypocalc.com> parents: 73358 diff changeset	1064 res = PyUnicode_FromOrdinal(ch);
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c Victor Stinner <victor.stinner@haypocalc.com> parents: 73358 diff changeset	1065 if (res == NULL)
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c Victor Stinner <victor.stinner@haypocalc.com> parents: 73358 diff changeset	1066 return NULL;
87261 0d9624f2ff43 Issue #12892: The utf-16* and utf-32* codecs now reject (lone) surrogates. Serhiy Storchaka <storchaka@gmail.com> parents: 87109 diff changeset	1067 return Py_BuildValue("(Nn)", res, start + bytelength);
53681 52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	1068 }
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	1069 else {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1070 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1071 return NULL;
53681 52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	1072 }
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	1073 }
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	1074
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1075 static PyObject *
53966 7c29bfbfd352 Rename utf8b error handler to surrogateescape. Martin v. Löwis <martin@v.loewis.de> parents: 53965 diff changeset	1076 PyCodec_SurrogateEscapeErrors(PyObject *exc)
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1077 {
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1078 PyObject *restuple;
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1079 PyObject *object;
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	1080 Py_ssize_t i;
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1081 Py_ssize_t start;
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1082 Py_ssize_t end;
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1083 PyObject *res;
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1084 if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1085 char *outp;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1086 if (PyUnicodeEncodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1087 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1088 if (PyUnicodeEncodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1089 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1090 if (!(object = PyUnicodeEncodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1091 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1092 res = PyBytes_FromStringAndSize(NULL, end-start);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1093 if (!res) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1094 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1095 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1096 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1097 outp = PyBytes_AsString(res);
73350 0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	1098 for (i = start; i < end; i++) {
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	1099 /* object is guaranteed to be "ready" */
0883d6a103a0 Port error handlers from Py_UNICODE indexing to code point indexing. Martin v. Löwis <martin@v.loewis.de> parents: 72925 diff changeset	1100 Py_UCS4 ch = PyUnicode_READ_CHAR(object, i);
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1101 if (ch < 0xdc80 \|\| ch > 0xdcff) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1102 /* Not a UTF-8b surrogate, fail with original exception */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1103 PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1104 Py_DECREF(res);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1105 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1106 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1107 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1108 *outp++ = ch - 0xdc00;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1109 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1110 restuple = Py_BuildValue("(On)", res, end);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1111 Py_DECREF(res);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1112 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1113 return restuple;
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1114 }
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1115 else if (PyObject_IsInstance(exc, PyExc_UnicodeDecodeError)) {
73359 e3b6458056c7 Avoid the Py_UNICODE type in codecs.c Victor Stinner <victor.stinner@haypocalc.com> parents: 73358 diff changeset	1116 PyObject *str;
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1117 unsigned char *p;
73359 e3b6458056c7 Avoid the Py_UNICODE type in codecs.c Victor Stinner <victor.stinner@haypocalc.com> parents: 73358 diff changeset	1118 Py_UCS2 ch[4]; /* decode up to 4 bad bytes. */
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1119 int consumed = 0;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1120 if (PyUnicodeDecodeError_GetStart(exc, &start))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1121 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1122 if (PyUnicodeDecodeError_GetEnd(exc, &end))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1123 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1124 if (!(object = PyUnicodeDecodeError_GetObject(exc)))
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1125 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1126 if (!(p = (unsigned char*)PyBytes_AsString(object))) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1127 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1128 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1129 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1130 while (consumed < 4 && consumed < end-start) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1131 /* Refuse to escape ASCII bytes. */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1132 if (p[start+consumed] < 128)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1133 break;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1134 ch[consumed] = 0xdc00 + p[start+consumed];
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1135 consumed++;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1136 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1137 Py_DECREF(object);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1138 if (!consumed) {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1139 /* codec complained about ASCII byte. */
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1140 PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1141 return NULL;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1142 }
73359 e3b6458056c7 Avoid the Py_UNICODE type in codecs.c Victor Stinner <victor.stinner@haypocalc.com> parents: 73358 diff changeset	1143 str = PyUnicode_FromKindAndData(PyUnicode_2BYTE_KIND, ch, consumed);
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c Victor Stinner <victor.stinner@haypocalc.com> parents: 73358 diff changeset	1144 if (str == NULL)
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c Victor Stinner <victor.stinner@haypocalc.com> parents: 73358 diff changeset	1145 return NULL;
e3b6458056c7 Avoid the Py_UNICODE type in codecs.c Victor Stinner <victor.stinner@haypocalc.com> parents: 73358 diff changeset	1146 return Py_BuildValue("(Nn)", str, start+consumed);
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1147 }
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1148 else {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1149 wrong_exception_type(exc);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1150 return NULL;
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1151 }
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1152 }
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1153
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1154
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1155 static PyObject strict_errors(PyObject self, PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1156 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1157 return PyCodec_StrictErrors(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1158 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1159
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1160
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1161 static PyObject ignore_errors(PyObject self, PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1162 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1163 return PyCodec_IgnoreErrors(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1164 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1165
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1166
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1167 static PyObject replace_errors(PyObject self, PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1168 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1169 return PyCodec_ReplaceErrors(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1170 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1171
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1172
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1173 static PyObject xmlcharrefreplace_errors(PyObject self, PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1174 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1175 return PyCodec_XMLCharRefReplaceErrors(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1176 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1177
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1178
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1179 static PyObject backslashreplace_errors(PyObject self, PyObject *exc)
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1180 {
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1181 return PyCodec_BackslashReplaceErrors(exc);
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1182 }
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1183
53965 fbcdae3e0ca1 Rename the surrogates error handler to surrogatepass. Martin v. Löwis <martin@v.loewis.de> parents: 53767 diff changeset	1184 static PyObject surrogatepass_errors(PyObject self, PyObject *exc)
53681 52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	1185 {
53965 fbcdae3e0ca1 Rename the surrogates error handler to surrogatepass. Martin v. Löwis <martin@v.loewis.de> parents: 53767 diff changeset	1186 return PyCodec_SurrogatePassErrors(exc);
53681 52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	1187 }
52a77ef069cd Issue #3672: Reject surrogates in utf-8 codec; add surrogates error Martin v. Löwis <martin@v.loewis.de> parents: 49613 diff changeset	1188
53966 7c29bfbfd352 Rename utf8b error handler to surrogateescape. Martin v. Löwis <martin@v.loewis.de> parents: 53965 diff changeset	1189 static PyObject surrogateescape_errors(PyObject self, PyObject *exc)
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1190 {
53966 7c29bfbfd352 Rename utf8b error handler to surrogateescape. Martin v. Löwis <martin@v.loewis.de> parents: 53965 diff changeset	1191 return PyCodec_SurrogateEscapeErrors(exc);
53767 ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1192 }
ac209f132b17 Issue #5915: Implement PEP 383, Non-decodable Bytes in Martin v. Löwis <martin@v.loewis.de> parents: 53687 diff changeset	1193
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1194 static int _PyCodecRegistry_Init(void)
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	1195 {
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1196 static struct {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1197 char *name;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1198 PyMethodDef def;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1199 } methods[] =
0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1200 {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1201 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1202 "strict",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1203 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1204 "strict_errors",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1205 strict_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1206 METH_O,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1207 PyDoc_STR("Implements the 'strict' error handling, which "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1208 "raises a UnicodeError on coding errors.")
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1209 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1210 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1211 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1212 "ignore",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1213 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1214 "ignore_errors",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1215 ignore_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1216 METH_O,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1217 PyDoc_STR("Implements the 'ignore' error handling, which "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1218 "ignores malformed data and continues.")
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1219 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1220 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1221 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1222 "replace",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1223 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1224 "replace_errors",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1225 replace_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1226 METH_O,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1227 PyDoc_STR("Implements the 'replace' error handling, which "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1228 "replaces malformed data with a replacement marker.")
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1229 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1230 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1231 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1232 "xmlcharrefreplace",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1233 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1234 "xmlcharrefreplace_errors",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1235 xmlcharrefreplace_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1236 METH_O,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1237 PyDoc_STR("Implements the 'xmlcharrefreplace' error handling, "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1238 "which replaces an unencodable character with the "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1239 "appropriate XML character reference.")
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1240 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1241 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1242 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1243 "backslashreplace",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1244 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1245 "backslashreplace_errors",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1246 backslashreplace_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1247 METH_O,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1248 PyDoc_STR("Implements the 'backslashreplace' error handling, "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1249 "which replaces an unencodable character with a "
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1250 "backslashed escape sequence.")
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1251 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1252 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1253 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1254 "surrogatepass",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1255 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1256 "surrogatepass",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1257 surrogatepass_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1258 METH_O
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1259 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1260 },
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1261 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1262 "surrogateescape",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1263 {
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1264 "surrogateescape",
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1265 surrogateescape_errors,
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1266 METH_O
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1267 }
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1268 }
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1269 };
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1270
31797 846b3ee7e106 Changed random calls to PyThreadState_Get() to use the macro Nicholas Bastin <nick.bastin@gmail.com> parents: 31273 diff changeset	1271 PyInterpreterState *interp = PyThreadState_GET()->interp;
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1272 PyObject *mod;
32533 201986d6d9d6 Fix a couple of signed/unsigned comparison warnings Neal Norwitz <nnorwitz@gmail.com> parents: 31797 diff changeset	1273 unsigned i;
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1274
850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1275 if (interp->codec_search_path != NULL)
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1276 return 0;
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1277
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1278 interp->codec_search_path = PyList_New(0);
850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1279 interp->codec_search_cache = PyDict_New();
850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1280 interp->codec_error_registry = PyDict_New();
850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1281
850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1282 if (interp->codec_error_registry) {
72510 b534b56379cb Use the new Py_ARRAY_LENGTH macro Victor Stinner <victor.stinner@haypocalc.com> parents: 72475 diff changeset	1283 for (i = 0; i < Py_ARRAY_LENGTH(methods); ++i) {
81031 3a86a3f1d89a Issue #15422: get rid of PyCFunction_New macro Andrew Svetlov <andrew.svetlov@gmail.com> parents: 80218 diff changeset	1284 PyObject *func = PyCFunction_NewEx(&methods[i].def, NULL, NULL);
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1285 int res;
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1286 if (!func)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1287 Py_FatalError("can't initialize codec error registry");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1288 res = PyCodec_RegisterError(methods[i].name, func);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1289 Py_DECREF(func);
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1290 if (res)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1291 Py_FatalError("can't initialize codec error registry");
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1292 }
25091 0337dad8403e PEP 293 implemention (from SF patch http://www.python.org/sf/432401) Walter Dörwald <walter@livinglogic.de> parents: 24307 diff changeset	1293 }
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1294
850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1295 if (interp->codec_search_path == NULL \|\|
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1296 interp->codec_search_cache == NULL \|\|
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1297 interp->codec_error_registry == NULL)
cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1298 Py_FatalError("can't initialize codec registry");
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	1299
44442 fa2f7183b1af More PyImport_ImportModule -> PyImport_ImportModuleNoBlock Christian Heimes <christian@cheimes.de> parents: 44351 diff changeset	1300 mod = PyImport_ImportModuleNoBlock("encodings");
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1301 if (mod == NULL) {
61466 cab14be0ada1 Recorded merge of revisions 81029 via svnmerge from Antoine Pitrou <solipsis@pitrou.net> parents: 56894 diff changeset	1302 return -1;
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1303 }
850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1304 Py_DECREF(mod);
49613 efef5fc45a9a Issue 3723: Fixed initialization of subinterpreters Christian Heimes <christian@cheimes.de> parents: 47496 diff changeset	1305 interp->codecs_initialized = 1;
28325 850626d47574 Fixed SF bug #663074. The codec system was using global static Gustavo Niemeyer <gustavo@niemeyer.net> parents: 27898 diff changeset	1306 return 0;
11972 d0e06efb3165 Python Codec Registry and support functions, written by Marc-Andre Guido van Rossum <guido@python.org> parents: diff changeset	1307 }

Mercurial > cpython-lru_cache-weakref

annotate Python/codecs.c @ 87690:66c1c9f32567 default tip